SlideShare a Scribd company logo
1 of 93
Download to read offline
Cloud Frontiers: A Deep
Dive into Serverless
Spatial Data and FME
Dean
Hintz
Technical Support Team
Lead, Strategic Solutions
Safe Software
Kailin
Opaleychuk
Technical Support Specialist,
FME Form
Safe Software
Welcome to Livestorm.
A few ways to engage with us during the webinar:
Audio issues? Click this for 4 simple
troubleshooting steps.
Agenda
1 Introduction
2 What is Cloud Native?
3 Perspectives from Radiant Earth, Planet
4 STAC & COGs
5 FlatGeoBuf
6 COPC & Zarr
7 GeoParquet
8 Lessons learned
9 Q&A
Agenda
Poll:
What’s your leading motive for
considering the use of cloud native?
1
Introduction
W
● Cloud native formats = cloud-optimized
● Specifically designed to optimize the storage, access, and processing of
geospatial data in cloud computing environments
● Supports data chunking, indexing, tiling and targeted metadata to minimize
response footprint
● Optimize access by thin web clients, whether browser or mobile base
● Partial & Parallel reads
● Read just what you need
What does Cloud-Native Mean?
Introduction
● Lazy access and intelligent subsetting
● Integrates well with high level analysis and distributed systems
● Scalable performance - increases the applicability of cloud-scale tools
● Decreases the burden and costs for data providers
● Tailored to leverage the scalability, flexibility, and processing power of cloud
infrastructure, enabling efficient handling of large data volumes.
Article: Cloud native data formats
Benefits of Cloud-Optimized Data
Introduction
Data Inspector COG Example: Canada DEM
COG Canada DEM on S3: full width, lowest zoom by default
Search envelope & CRS needed to limit request:
● CRS of dataset on server
● Search envelop CRS - same units as extent, can be
different than source dataset
● Pyramid level to read. In this case level 1 = 30m resolution
Data Inspector COG Example: Canada DEM
2 seconds to access DEM for all of Canada from 31GB COG source dataset
level 10 = 16 km x 16 km grid cells
Data Inspector COG Example: Canada DEM
4 seconds to access 30m DEM for all Fraser Valley from 31GB COG source dataset
31GB COG dataset - do not open in browser!
https://datacube-prod-data-public.s3.ca-central-1.amazonaws.com/store/elevation/cdem-cdsm/cdem/cdem-canada-dem.tif
Part 1
Recap
Chris Holmes
VP of Product, Strategy, Partnerships
Planet
Michelle Roby
Developer Advocate
Radiant Earth
Cape Town, South Africa • March 19, 2017
Planet / Cloud Native Geo Foundation / Taylor Geospatial Engine
Cloud Native Geospatial Origins
Chris Holmes
About Radiant Earth
About:
● An incubator of data-driven initiatives, services, and 21st century institutions needed to
foster shared understanding of our world
Initiatives:
● Cloud-Native Geospatial Foundation → Aim to increase adoption of highly efficient
approaches to working with geospatial data on the Internet.
● Source Cooperative → Data publishing utility for easy data sharing over the web.
What makes cloud-optimized challenging?
From Task 51 Study:
“There is no
one-size-fits-all
packaging for data, as
the optimal packaging is
highly use-case
dependent.”
Authors: Chris Durbin, Patrick Quinn, Dana
Shum
New Cloud-Native Format Support
Format Support Version Available
Cloud Optimized Geotiff R / W 2023.0
Cloud Optimized Point Cloud R / W 2023.1 / 2023.2 (2024.0)
FlatGeoBuf R / W 2023.0
GeoParquet R / W 2023.1
SpatioTemporal Asset Catalog
(Metadata + Asset)
R 2024.0 (FME Hub)*
ZARR R / W 2023.1 (2024.0)
2
STAC
(SpatioTemporal
Asset Catalog)
● Spatio-Temporal Asset Catalog
is a format that stores cloud-based
assets that relate to a
geographic area or time.
● The assets are templated in a
JSON catalog/collection.
● Supports raster and vector
assets
○ For example, a STAC
Collection can have Assets
that store geopackage layers
or COG bands as asset
items.
STAC
STAC Package (FME Hub)
- STAC Package V2.1.1 now available on the FME Hub.
- STAC Metadata Reader*
- STAC Asset Reader
- V2.0.0 requires FME 24.0 minimum build 24134
STAC Metadata Reader
Images demonstrating how to use
the STAC Metadata Reader to dig
down into a STAC Collection
https://spot-canada-ortho.s3.amazonaws.com/catalog.json
Catalog > Collection > Item > Asset
Slide Title
Consume a
GeoTIFF in
STAC and
convert to Cloud
Optimized
GeoTIFF
Goal Key Result
Working with STAC Asset Reader in FME Form
Use the FME
platform to refine
and translate data
from one location
to another
Output Cloud
Optimized
Geotiff ready for
further analysis
on S3
Demo
● Use raster transformers to post-process STAC assets
○ Combining raster bands
○ Setting & removing no data
● FME’s S3Connector can publish COGs to the cloud
Demo Summary
Removing no data
FME Form Workspace
Demo Results
FME & STAC Overview
● One set of item assets can be read or
accessed by a single reader feature type
● STAC Metadata Reader can be used to
filter and select assets of interest
● Coming Soon
○ The ability to access authentication
required assets
○ Pre-defined popular STAC API
options to improve usability
3
COGs
(Cloud Optimized
GeoTIFFs)
● Supports raster data
● Built off the GeoTIFF
specification, which offers
functionality for
compression and
pyramiding
● Benefits from partial reads,
through MinMax extents or
clipping.
COG
COG Reader
● Search Envelope
● Pyramid level options
COG Writer
● Writer feature type
○ Compression
○ Layout: Cloud
Optimized Tiles
○ Pyramid level
options
COG Reader in FME Form
https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif
COG Reader - Search Envelope
Reading entire dataset
Reading with Search Envelope constraint
Slide Title
Create an
insightful report
on recent fires
West of Kelowna
Goal Key Result
Current Fire Mapping for West Kelowna
Use transformers
to extract, combine
& reformat data
An interactive
HTML report
with embedded
images and links
Demo
Demo Results
● FlatGeoBuf and COG readers support
spatial filter operations
● Use polygon mask to refine points on
Nodata areas
● XMLTemplater can be used to help format
HTML elements, such as tables
Key Demo Takeaways
4
FlatGeoBuf
● Vector format built on
Google’s Flatbuffers library
● A buffer is considered a file
and everything within it.
● Although it is not required,
FlatGeobuf uses indexing to
help reduce the amount of
data that would need to be
transferred over a potentially
slow network.
FlatGeoBuf
FlatGeoBuf Reader
● Verify file buffers
● Search envelope
FlatGeoBuf Writer
● Create spatial index
Slide Title
Create a service that
automatically
uploads a range of
vector data to S3 as
FlatGeoBuf
Goal Key Result
FlatGeoBuf S3 Uploader App
Generic Reader
paired with user
parameters
Uploaded
buffers and an
upload html
upload report
Demo
● User parameters help make workspaces
more dynamic
● PROJReprojector with online grids
enabled
Summary of FlatGeoBuf S3
Uploader App
5
COPC
(Cloud Optimized
Point Cloud)
● Point cloud storage
optimized for the web
● Based on the LAS standard
● Only read what you need.
This is especially powerful for
point clouds given 3d data
data volumes can be huge
● Query XY min/max
● Essentially uses the LAS
reader / writer but with the
COPC structure
COPC
● Point cloud
generated from
drone imagery
using dense point
matching: ODM
● 1.1 million points
● Converted from
LAS to COPC and
loaded to S3
COPC - White Rock Pier Post Storm from Drone Survey
● Uses the LAS reader / writer
but with the COPC structure
● LAZ compression
● Select Write as Cloud
Optimized Point Cloud
● Set CRS
COPC Writing
● Use S3Loader to upload
COPC to S3 bucket
● Compressed vs
uncompressed tradeoff
COPC S3 Loader
COPC Reading: Convert PointCloud to Polygon Features
1. Read COPC from S3
2. Filter out ground
points
3. Convert to raster
4. Classify raster
5. Convert raster to
polygons
6. Filter out donuts &
small polygons
7. Write result to
geojson
Read only points close to pier
COPC Reading - Extents filter
Convert pier point cloud to areas, calculate distance of collapse (47m)
COPC Reading: Pier Polygons Written to GeoJSON
● Point cloud storage
optimized for the web
● 2024.0 fixes and
enhancements (round trip)
● Extents query supported and
optimized
● Coming: FeatureReader
clipping by extents
COPC Summary
6
ZARR
Format
● Multidimensional raster array /
time series storage optimized for
the web
● Based on NetCDF / HDF data
cube formats
● Only read what you need
● Particularly powerful for raster
time series, as multidimensional
arrays often mean huge volumes
● Query XY& band* extents
● Zarr reads cube with each time
step as a separate band with
properties - easy to work with
ZARR
● Time series raster storage
optimized for the web
● Based on NetCDF data cube
● NetCDF reads cube as multigrid
with 1 band for each time step
(hundreds of bands) and
properties in attribute lists
● Zarr reads cube with each time
step as a separate band with
properties - easier to work with
● Default translation from NetCDF
to Zarr just works*
NetCDF to ZARR
ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB
ZARR CMIP5 Climate Model Temp Analysis: Hotspots in Winnipeg, MB
ZARR Climate Model Band Range Request: Python Parameters
ZARR CMIP5 Climate Model Temp Analysis: Hotspots in Winnipeg, MB
OGC Climate Resilience Pilot 2023
Pilot Goals:
● Build climate resilience
● Expand audience for climate
services
● Demonstrate the value of OGC
standards and SDI’s (FAIR)
● Show how OGC can support
international climate change goals
● Build a community of stakeholders
better understand the range of possible
impacts - allows us to better prepare and
compensate for them
https://www.ogc.org/initiatives/crp/
How to provide the data needed for climate impact and
disaster indicators to a wider audience?
● Goal: Connect Climate and Disaster Pilots
● Data: Current situational awareness
○ Base map: physical, land use, infrastructure, pop
○ EO data: hazards and impacts
○ Drought & hydrologic monitoring
● Data: Future change awareness - risk scenarios due to
climate change
○ Climate model outputs - time series data cubes
○ Temperature, precipitation and moisture projections
○ Analysis Ready Data (ARD) model results summary
○ Climate services known in climate community but not well
known or utilized across affected impact domains
NetCDF from Environment Canada
Disaster Pilot 2023:
Disaster and Climate Data Sources to ARD & Impacts
MB Drought Risk: Combined Precip Temp Query
OGC API Features Query Parameters:
Start Year: 2020
End Year: 2060
BBox: -100.0,49.0,-96.0,50.5
Limit: 2,000,000
MinPeriodValue: 0 (PrecipDelta)
MaxPeriodValue: 0.75 (PrecipDelta)
MinTemp: 23C (Min Mean Monthly Temp)
Find all time step points over the next 40
years for southern Manitoba where
projections indicate:
● > 25% dryer than historical mean
AND
● mean monthly temperature > 23C
MB Precipitation: Future Delta
PrecipDelta = PrecipFuture / PrecipHistoricalMean
/
=
Yields normalized value from 0 to N where 0 = no precipitation and 1.0 = 100% of historical mean
MB Drought Risk: Combined Precip Temp Output
● Multidimensional raster array / time series
storage optimized for the web
● Based on NetCDF / HDF data cube formats
● Only read what you need
● Zarr reads cube with each time step as a
separate band
● Query XYextents
● Band range - supports max not min
● Be careful with feature cache
● Data Inspector refresh needed with stacked
rasters: select range and then select cell again
ZARR Summary
7
GeoParquet
GeoParquet
● Cloud native / cloud friendly vector data storage
● Built on & follows Parquet standards
● Column oriented
● Highly optimized for accessing very large data
volumes where you need access to a few columns
and geometry, such as for analysis
● Benefits from a mature set of applications, libraries
& tools available for Parquet
● Supports a range of geometries
● Not spatially indexed yet (use partitioning, duckDB)
Slide Title
Optimize reading
and analysis of
published large
vector dataset
Goal Block Key
GeoParquet reader performance demo
Result
Internet
bandwidth and
local processing
limitations
Structure data so
you only read
what you need
Test case:
Geoparquet is 2 - 3
X faster than other
alternatives
Demo
Performance: Geoparquet vs OSM, Geopackage
1 millions records, select and spatially analyze 107k water areas
GeoParquet Partitioning
Nested structure with folder by feature type and
separate files for each value for selected attribute
GeoParquet Partitioning
Only read the features with the
feature type and values you want
Nested structure with folders by
feature type and separate files for
each value for selected attribute
Reader Local S3 Cloud -> local S3 Cloud -> FME Hosted
OSM reporter* 23.2 60.4 38.1
Geopackage
reporter*
1.2 102.8 14
GeoParquet
reporter*
1.3 37.5 7.2
GeoParquet
partitioned*
0.3 15.2 4.9
Performance: Geoparquet vs OSM, Geopackage
*1 millions records, select and spatially analyze 100k
water areas. Process time in seconds
● Column oriented vector format
● Geoparquet test: 2-3X faster than
others
● Cloud native for vector not as easy
as for raster, point cloud
● Adds requirement for appropriate
cataloging
● Additional speed improvements
with more attribute level partitioning
● This addresses some of the debate
around geoparquet as cloud native
● DuckDB with Geoparquet to
improve cloud native performance
Key Lessons
GeoParquet
Other Cloud Data Stores: Cloud Databases
… to name a few
8
Key limitations
& Integration
Strategies
● Start publishing now!
● Keep the processing close to the data
● Minimize traffic footprint - select just what you need
● Leverage data side filtering, microservices, lazy evaluation
● Metadata: enrich and update
● Optimization strategy: transactions volume vs data volume, response time requirements
● Test! Especially your core usage scenarios
Integration Strategies
Key limitations & Integration Strategies
Considerations
● Heavier preprocessing, larger size required to structure and store data for optimized read
● Updates are a challenge - automation helps
● FME’s implementation based on third party libraries - collaboration for fixes,
enhancements
● Newer cloud native formats: less data publicly available so far: COPC, ZARR
● Cloud optimized vector options - choice depend on use case: GeoParquet, FlatGeoBuf
● Supporting infrastructure: duckDB for Geoparquet etc
Key limitations & Integration Strategies
Geoparquet & FlatGeoBuf
Yes
9
Conclusions
Optimize
your web
data flows
Lessons Learned
● Cloud native is all about making it easy to publish data
without a server, optimizing responses to web data
requests: read just what you need!
● No one size fits all: each format has its strengths and
limitations
● STAC: steeper learning curve, collections within
catalogs and vice versa, implementations vary; security
● COG, COPC: perhaps the most intuitive - 2d and 3d
arrays are just easier to manage. STAC/COG most data
● Vector - still evolving: FlatGeoBuf more effective in its
cloud native support but newer, less widely adopted.
Geoparquet has more tooling but design and support
needed to make effectively cloud native
● ZARR - powerful but complex - as a very new and niche
format, support is still growing
Summary
● Cloud native is all about making it easy to publish
data without a server, optimizing responses to
web data requests: read just what you need!
● Safe’s strategy is to track and support emerging
standards across a range of data types so FME
users can stay ahead of evolving web technologies
● FME allows you to integrate between hybrid
environments as needed
● Keep the processing close to the data
● Minimize traffic footprint - reader filtering
● Open standards enable community-wide adoption
and access
● No one size fits all - know your key requirements &
test!
One platform, two technologies
FME Form FME Flow
Build and run data workflows Automate data workflows
FME Flow Hosted
Safe Software managed instance
fme.safe.com/platform
FME Enterprise Integration Platform
Safe & FME
10
Resources
Resources
● Geospatial Cloud Native Overview
● Webinar: Cloud Revolution: Exploring the New Wave of
Serverless Spatial Data
● Webinar: Leveraging FME for Cloud Native Databases
● Cloud Native Databases - Blog
● Radiant Earth Blog: Cloud Native Geospatial Solutions
● Cloud Native Geospatial Foundation & Slack channel
● OGC - Cloud Native Geospatial
● Source Cooperative
● Chris Holmes: FOSS4G NA 2023 | Towards a Cloud
Native Spatial Data Infrastructure
● guide.cloudnativegeo.org
● Cloud Native Data Formats
● Safe’s Participation in OGC Pilots
Data Sources
STAC / COG:
● catalogue.dataspace.copernicus.eu/stac/
● cmr.earthdata.nasa.gov/stac/
● planetarycomputer.microsoft.com/catalog
● usgs.gov/landsat-missions/landsat-collection-2
● planetarycomputer.microsoft.com/api/stac/v1/colle
ctions/sentinel-2-l2a
● https://planetarycomputer.microsoft.com/api/stac/
v1/collections/nrcan-landcover
ZARR:
● https://console.cloud.google.com/marketplace/pro
duct/noaa-public/cmip6
COPC:
● github.com/PDAL/data/tree/master/autzen
● copc.io/#example-data
11
Next Steps
Next Steps
● Coming:
○ Blogs
○ Tutorials
● Community involvement: Cloud Native
Geospatial Foundation, OGC
● Events:
○ cncf.io/events/
○ safe.com/company-updates/2024-upcomi
ng-events/
○ ogcmeet.org/
● New functionality: what are your priorities?
○ DuckDB?
○ ZARR band range?
○ Other STAC media types?
Get our Ebook
Spatial Data for the
Enterprise
fme.ly/gzc
Guided learning
experiences at your
fingertips
academy.safe.com
FME Academy
Resources
Check out how-to’s &
demos in the knowledge
base
community.safe.com
/s/knowledge-base
Knowledge Base Webinars
Upcoming &
on-demand webinars
safe.com/webinars
ClaimYour Community Badge
● Get community badges for watching
webinars!
● fme.ly/WebinarBadge
● Today’s code: SLMWB
Join the Community today!
12
Q&A
ThankYou
Recap of Next Steps
1 Contact us: info@safe.com
2 Check out the new landing page to get started!
Please fill out our
webinar survey

More Related Content

Similar to Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

How @twitterhadoop chose google cloud
How @twitterhadoop chose google cloudHow @twitterhadoop chose google cloud
How @twitterhadoop chose google cloudlohitvijayarenu
 
State of GeoServer - FOSS4G 2016
State of GeoServer - FOSS4G 2016State of GeoServer - FOSS4G 2016
State of GeoServer - FOSS4G 2016GeoSolutions
 
Cloud Composer workshop at Airflow Summit 2023.pdf
Cloud Composer workshop at Airflow Summit 2023.pdfCloud Composer workshop at Airflow Summit 2023.pdf
Cloud Composer workshop at Airflow Summit 2023.pdfLeah Cole
 
Get Your Head in the Cloud - Lessons in GPU Computing with Schlumberger
Get Your Head in the Cloud - Lessons in GPU Computing with SchlumbergerGet Your Head in the Cloud - Lessons in GPU Computing with Schlumberger
Get Your Head in the Cloud - Lessons in GPU Computing with Schlumbergerinside-BigData.com
 
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...NETWAYS
 
Apache Beam and Google Cloud Dataflow - IDG - final
Apache Beam and Google Cloud Dataflow - IDG - finalApache Beam and Google Cloud Dataflow - IDG - final
Apache Beam and Google Cloud Dataflow - IDG - finalSub Szabolcs Feczak
 
Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding HadoopAhmed Ossama
 
Session 8 - Creating Data Processing Services | Train the Trainers Program
Session 8 - Creating Data Processing Services | Train the Trainers ProgramSession 8 - Creating Data Processing Services | Train the Trainers Program
Session 8 - Creating Data Processing Services | Train the Trainers ProgramFIWARE
 
AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...
AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...
AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...Amazon Web Services
 
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationMaximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationDenodo
 
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent
 
Big data Argentina meetup 2020-09: Intro to presto on docker
Big data Argentina meetup 2020-09: Intro to presto on dockerBig data Argentina meetup 2020-09: Intro to presto on docker
Big data Argentina meetup 2020-09: Intro to presto on dockerFederico Palladoro
 
NetflixOSS Meetup season 3 episode 1
NetflixOSS Meetup season 3 episode 1NetflixOSS Meetup season 3 episode 1
NetflixOSS Meetup season 3 episode 1Ruslan Meshenberg
 
Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...
Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...
Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...DataStax
 
Solving enterprise challenges through scale out storage & big compute final
Solving enterprise challenges through scale out storage & big compute finalSolving enterprise challenges through scale out storage & big compute final
Solving enterprise challenges through scale out storage & big compute finalAvere Systems
 
Druid at naver.com - part 1
Druid at naver.com - part 1Druid at naver.com - part 1
Druid at naver.com - part 1Jungsu Heo
 
Big Data on Cloud Native Platform
Big Data on Cloud Native PlatformBig Data on Cloud Native Platform
Big Data on Cloud Native PlatformSunil Govindan
 
Big Data on Cloud Native Platform
Big Data on Cloud Native PlatformBig Data on Cloud Native Platform
Big Data on Cloud Native PlatformSunil Govindan
 
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudExtending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudDataWorks Summit
 

Similar to Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME (20)

How @twitterhadoop chose google cloud
How @twitterhadoop chose google cloudHow @twitterhadoop chose google cloud
How @twitterhadoop chose google cloud
 
State of GeoServer - FOSS4G 2016
State of GeoServer - FOSS4G 2016State of GeoServer - FOSS4G 2016
State of GeoServer - FOSS4G 2016
 
Cloud Composer workshop at Airflow Summit 2023.pdf
Cloud Composer workshop at Airflow Summit 2023.pdfCloud Composer workshop at Airflow Summit 2023.pdf
Cloud Composer workshop at Airflow Summit 2023.pdf
 
Get Your Head in the Cloud - Lessons in GPU Computing with Schlumberger
Get Your Head in the Cloud - Lessons in GPU Computing with SchlumbergerGet Your Head in the Cloud - Lessons in GPU Computing with Schlumberger
Get Your Head in the Cloud - Lessons in GPU Computing with Schlumberger
 
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
 
Apache Beam and Google Cloud Dataflow - IDG - final
Apache Beam and Google Cloud Dataflow - IDG - finalApache Beam and Google Cloud Dataflow - IDG - final
Apache Beam and Google Cloud Dataflow - IDG - final
 
Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding Hadoop
 
Session 8 - Creating Data Processing Services | Train the Trainers Program
Session 8 - Creating Data Processing Services | Train the Trainers ProgramSession 8 - Creating Data Processing Services | Train the Trainers Program
Session 8 - Creating Data Processing Services | Train the Trainers Program
 
AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...
AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...
AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...
 
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationMaximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
 
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
 
Big data Argentina meetup 2020-09: Intro to presto on docker
Big data Argentina meetup 2020-09: Intro to presto on dockerBig data Argentina meetup 2020-09: Intro to presto on docker
Big data Argentina meetup 2020-09: Intro to presto on docker
 
NetflixOSS Meetup season 3 episode 1
NetflixOSS Meetup season 3 episode 1NetflixOSS Meetup season 3 episode 1
NetflixOSS Meetup season 3 episode 1
 
Data Platform on GCP
Data Platform on GCPData Platform on GCP
Data Platform on GCP
 
Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...
Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...
Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...
 
Solving enterprise challenges through scale out storage & big compute final
Solving enterprise challenges through scale out storage & big compute finalSolving enterprise challenges through scale out storage & big compute final
Solving enterprise challenges through scale out storage & big compute final
 
Druid at naver.com - part 1
Druid at naver.com - part 1Druid at naver.com - part 1
Druid at naver.com - part 1
 
Big Data on Cloud Native Platform
Big Data on Cloud Native PlatformBig Data on Cloud Native Platform
Big Data on Cloud Native Platform
 
Big Data on Cloud Native Platform
Big Data on Cloud Native PlatformBig Data on Cloud Native Platform
Big Data on Cloud Native Platform
 
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudExtending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google Cloud
 

More from Safe Software

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemSafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Safe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Safe Software
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMESafe Software
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Safe Software
 

More from Safe Software (20)

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FME
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...
 

Recently uploaded

Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireExakis Nelite
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxJennifer Lim
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyJohn Staveley
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfSrushith Repakula
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...CzechDreamin
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...FIDO Alliance
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...marcuskenyatta275
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftshyamraj55
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfFIDO Alliance
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandIES VE
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceSamy Fodil
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024Stephanie Beckett
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyUXDXConf
 

Recently uploaded (20)

Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

  • 1. Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
  • 2. Dean Hintz Technical Support Team Lead, Strategic Solutions Safe Software Kailin Opaleychuk Technical Support Specialist, FME Form Safe Software
  • 3. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
  • 4. Agenda 1 Introduction 2 What is Cloud Native? 3 Perspectives from Radiant Earth, Planet 4 STAC & COGs 5 FlatGeoBuf 6 COPC & Zarr 7 GeoParquet 8 Lessons learned 9 Q&A Agenda
  • 5. Poll: What’s your leading motive for considering the use of cloud native?
  • 7. W ● Cloud native formats = cloud-optimized ● Specifically designed to optimize the storage, access, and processing of geospatial data in cloud computing environments ● Supports data chunking, indexing, tiling and targeted metadata to minimize response footprint ● Optimize access by thin web clients, whether browser or mobile base ● Partial & Parallel reads ● Read just what you need What does Cloud-Native Mean? Introduction
  • 8. ● Lazy access and intelligent subsetting ● Integrates well with high level analysis and distributed systems ● Scalable performance - increases the applicability of cloud-scale tools ● Decreases the burden and costs for data providers ● Tailored to leverage the scalability, flexibility, and processing power of cloud infrastructure, enabling efficient handling of large data volumes. Article: Cloud native data formats Benefits of Cloud-Optimized Data Introduction
  • 9. Data Inspector COG Example: Canada DEM COG Canada DEM on S3: full width, lowest zoom by default Search envelope & CRS needed to limit request: ● CRS of dataset on server ● Search envelop CRS - same units as extent, can be different than source dataset ● Pyramid level to read. In this case level 1 = 30m resolution
  • 10. Data Inspector COG Example: Canada DEM 2 seconds to access DEM for all of Canada from 31GB COG source dataset level 10 = 16 km x 16 km grid cells
  • 11. Data Inspector COG Example: Canada DEM 4 seconds to access 30m DEM for all Fraser Valley from 31GB COG source dataset 31GB COG dataset - do not open in browser! https://datacube-prod-data-public.s3.ca-central-1.amazonaws.com/store/elevation/cdem-cdsm/cdem/cdem-canada-dem.tif
  • 12. Part 1 Recap Chris Holmes VP of Product, Strategy, Partnerships Planet Michelle Roby Developer Advocate Radiant Earth
  • 13. Cape Town, South Africa • March 19, 2017 Planet / Cloud Native Geo Foundation / Taylor Geospatial Engine Cloud Native Geospatial Origins Chris Holmes
  • 14.
  • 15. About Radiant Earth About: ● An incubator of data-driven initiatives, services, and 21st century institutions needed to foster shared understanding of our world Initiatives: ● Cloud-Native Geospatial Foundation → Aim to increase adoption of highly efficient approaches to working with geospatial data on the Internet. ● Source Cooperative → Data publishing utility for easy data sharing over the web.
  • 16. What makes cloud-optimized challenging? From Task 51 Study: “There is no one-size-fits-all packaging for data, as the optimal packaging is highly use-case dependent.” Authors: Chris Durbin, Patrick Quinn, Dana Shum
  • 17. New Cloud-Native Format Support Format Support Version Available Cloud Optimized Geotiff R / W 2023.0 Cloud Optimized Point Cloud R / W 2023.1 / 2023.2 (2024.0) FlatGeoBuf R / W 2023.0 GeoParquet R / W 2023.1 SpatioTemporal Asset Catalog (Metadata + Asset) R 2024.0 (FME Hub)* ZARR R / W 2023.1 (2024.0)
  • 19. ● Spatio-Temporal Asset Catalog is a format that stores cloud-based assets that relate to a geographic area or time. ● The assets are templated in a JSON catalog/collection. ● Supports raster and vector assets ○ For example, a STAC Collection can have Assets that store geopackage layers or COG bands as asset items. STAC
  • 20. STAC Package (FME Hub) - STAC Package V2.1.1 now available on the FME Hub. - STAC Metadata Reader* - STAC Asset Reader - V2.0.0 requires FME 24.0 minimum build 24134
  • 21. STAC Metadata Reader Images demonstrating how to use the STAC Metadata Reader to dig down into a STAC Collection https://spot-canada-ortho.s3.amazonaws.com/catalog.json Catalog > Collection > Item > Asset
  • 22. Slide Title Consume a GeoTIFF in STAC and convert to Cloud Optimized GeoTIFF Goal Key Result Working with STAC Asset Reader in FME Form Use the FME platform to refine and translate data from one location to another Output Cloud Optimized Geotiff ready for further analysis on S3
  • 23. Demo
  • 24. ● Use raster transformers to post-process STAC assets ○ Combining raster bands ○ Setting & removing no data ● FME’s S3Connector can publish COGs to the cloud Demo Summary Removing no data FME Form Workspace
  • 26. FME & STAC Overview ● One set of item assets can be read or accessed by a single reader feature type ● STAC Metadata Reader can be used to filter and select assets of interest ● Coming Soon ○ The ability to access authentication required assets ○ Pre-defined popular STAC API options to improve usability
  • 28. ● Supports raster data ● Built off the GeoTIFF specification, which offers functionality for compression and pyramiding ● Benefits from partial reads, through MinMax extents or clipping. COG
  • 29. COG Reader ● Search Envelope ● Pyramid level options COG Writer ● Writer feature type ○ Compression ○ Layout: Cloud Optimized Tiles ○ Pyramid level options
  • 30. COG Reader in FME Form https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif
  • 31. COG Reader - Search Envelope Reading entire dataset Reading with Search Envelope constraint
  • 32. Slide Title Create an insightful report on recent fires West of Kelowna Goal Key Result Current Fire Mapping for West Kelowna Use transformers to extract, combine & reformat data An interactive HTML report with embedded images and links
  • 33. Demo
  • 35. ● FlatGeoBuf and COG readers support spatial filter operations ● Use polygon mask to refine points on Nodata areas ● XMLTemplater can be used to help format HTML elements, such as tables Key Demo Takeaways
  • 37. ● Vector format built on Google’s Flatbuffers library ● A buffer is considered a file and everything within it. ● Although it is not required, FlatGeobuf uses indexing to help reduce the amount of data that would need to be transferred over a potentially slow network. FlatGeoBuf
  • 38. FlatGeoBuf Reader ● Verify file buffers ● Search envelope FlatGeoBuf Writer ● Create spatial index
  • 39. Slide Title Create a service that automatically uploads a range of vector data to S3 as FlatGeoBuf Goal Key Result FlatGeoBuf S3 Uploader App Generic Reader paired with user parameters Uploaded buffers and an upload html upload report
  • 40. Demo
  • 41. ● User parameters help make workspaces more dynamic ● PROJReprojector with online grids enabled Summary of FlatGeoBuf S3 Uploader App
  • 43. ● Point cloud storage optimized for the web ● Based on the LAS standard ● Only read what you need. This is especially powerful for point clouds given 3d data data volumes can be huge ● Query XY min/max ● Essentially uses the LAS reader / writer but with the COPC structure COPC
  • 44. ● Point cloud generated from drone imagery using dense point matching: ODM ● 1.1 million points ● Converted from LAS to COPC and loaded to S3 COPC - White Rock Pier Post Storm from Drone Survey
  • 45. ● Uses the LAS reader / writer but with the COPC structure ● LAZ compression ● Select Write as Cloud Optimized Point Cloud ● Set CRS COPC Writing
  • 46. ● Use S3Loader to upload COPC to S3 bucket ● Compressed vs uncompressed tradeoff COPC S3 Loader
  • 47. COPC Reading: Convert PointCloud to Polygon Features 1. Read COPC from S3 2. Filter out ground points 3. Convert to raster 4. Classify raster 5. Convert raster to polygons 6. Filter out donuts & small polygons 7. Write result to geojson
  • 48. Read only points close to pier COPC Reading - Extents filter
  • 49. Convert pier point cloud to areas, calculate distance of collapse (47m) COPC Reading: Pier Polygons Written to GeoJSON
  • 50. ● Point cloud storage optimized for the web ● 2024.0 fixes and enhancements (round trip) ● Extents query supported and optimized ● Coming: FeatureReader clipping by extents COPC Summary
  • 52. ● Multidimensional raster array / time series storage optimized for the web ● Based on NetCDF / HDF data cube formats ● Only read what you need ● Particularly powerful for raster time series, as multidimensional arrays often mean huge volumes ● Query XY& band* extents ● Zarr reads cube with each time step as a separate band with properties - easy to work with ZARR
  • 53.
  • 54. ● Time series raster storage optimized for the web ● Based on NetCDF data cube ● NetCDF reads cube as multigrid with 1 band for each time step (hundreds of bands) and properties in attribute lists ● Zarr reads cube with each time step as a separate band with properties - easier to work with ● Default translation from NetCDF to Zarr just works* NetCDF to ZARR
  • 55. ZARR CMIP5 Climate Model Temp Analysis: Winnipeg, MB
  • 56. ZARR CMIP5 Climate Model Temp Analysis: Hotspots in Winnipeg, MB
  • 57. ZARR Climate Model Band Range Request: Python Parameters
  • 58. ZARR CMIP5 Climate Model Temp Analysis: Hotspots in Winnipeg, MB
  • 59. OGC Climate Resilience Pilot 2023 Pilot Goals: ● Build climate resilience ● Expand audience for climate services ● Demonstrate the value of OGC standards and SDI’s (FAIR) ● Show how OGC can support international climate change goals ● Build a community of stakeholders better understand the range of possible impacts - allows us to better prepare and compensate for them https://www.ogc.org/initiatives/crp/
  • 60. How to provide the data needed for climate impact and disaster indicators to a wider audience? ● Goal: Connect Climate and Disaster Pilots ● Data: Current situational awareness ○ Base map: physical, land use, infrastructure, pop ○ EO data: hazards and impacts ○ Drought & hydrologic monitoring ● Data: Future change awareness - risk scenarios due to climate change ○ Climate model outputs - time series data cubes ○ Temperature, precipitation and moisture projections ○ Analysis Ready Data (ARD) model results summary ○ Climate services known in climate community but not well known or utilized across affected impact domains NetCDF from Environment Canada Disaster Pilot 2023: Disaster and Climate Data Sources to ARD & Impacts
  • 61. MB Drought Risk: Combined Precip Temp Query OGC API Features Query Parameters: Start Year: 2020 End Year: 2060 BBox: -100.0,49.0,-96.0,50.5 Limit: 2,000,000 MinPeriodValue: 0 (PrecipDelta) MaxPeriodValue: 0.75 (PrecipDelta) MinTemp: 23C (Min Mean Monthly Temp) Find all time step points over the next 40 years for southern Manitoba where projections indicate: ● > 25% dryer than historical mean AND ● mean monthly temperature > 23C
  • 62. MB Precipitation: Future Delta PrecipDelta = PrecipFuture / PrecipHistoricalMean / = Yields normalized value from 0 to N where 0 = no precipitation and 1.0 = 100% of historical mean
  • 63. MB Drought Risk: Combined Precip Temp Output
  • 64. ● Multidimensional raster array / time series storage optimized for the web ● Based on NetCDF / HDF data cube formats ● Only read what you need ● Zarr reads cube with each time step as a separate band ● Query XYextents ● Band range - supports max not min ● Be careful with feature cache ● Data Inspector refresh needed with stacked rasters: select range and then select cell again ZARR Summary
  • 66. GeoParquet ● Cloud native / cloud friendly vector data storage ● Built on & follows Parquet standards ● Column oriented ● Highly optimized for accessing very large data volumes where you need access to a few columns and geometry, such as for analysis ● Benefits from a mature set of applications, libraries & tools available for Parquet ● Supports a range of geometries ● Not spatially indexed yet (use partitioning, duckDB)
  • 67. Slide Title Optimize reading and analysis of published large vector dataset Goal Block Key GeoParquet reader performance demo Result Internet bandwidth and local processing limitations Structure data so you only read what you need Test case: Geoparquet is 2 - 3 X faster than other alternatives
  • 68. Demo
  • 69. Performance: Geoparquet vs OSM, Geopackage 1 millions records, select and spatially analyze 107k water areas
  • 70. GeoParquet Partitioning Nested structure with folder by feature type and separate files for each value for selected attribute
  • 71. GeoParquet Partitioning Only read the features with the feature type and values you want Nested structure with folders by feature type and separate files for each value for selected attribute
  • 72. Reader Local S3 Cloud -> local S3 Cloud -> FME Hosted OSM reporter* 23.2 60.4 38.1 Geopackage reporter* 1.2 102.8 14 GeoParquet reporter* 1.3 37.5 7.2 GeoParquet partitioned* 0.3 15.2 4.9 Performance: Geoparquet vs OSM, Geopackage *1 millions records, select and spatially analyze 100k water areas. Process time in seconds
  • 73. ● Column oriented vector format ● Geoparquet test: 2-3X faster than others ● Cloud native for vector not as easy as for raster, point cloud ● Adds requirement for appropriate cataloging ● Additional speed improvements with more attribute level partitioning ● This addresses some of the debate around geoparquet as cloud native ● DuckDB with Geoparquet to improve cloud native performance Key Lessons GeoParquet
  • 74. Other Cloud Data Stores: Cloud Databases … to name a few
  • 76. ● Start publishing now! ● Keep the processing close to the data ● Minimize traffic footprint - select just what you need ● Leverage data side filtering, microservices, lazy evaluation ● Metadata: enrich and update ● Optimization strategy: transactions volume vs data volume, response time requirements ● Test! Especially your core usage scenarios Integration Strategies Key limitations & Integration Strategies
  • 77. Considerations ● Heavier preprocessing, larger size required to structure and store data for optimized read ● Updates are a challenge - automation helps ● FME’s implementation based on third party libraries - collaboration for fixes, enhancements ● Newer cloud native formats: less data publicly available so far: COPC, ZARR ● Cloud optimized vector options - choice depend on use case: GeoParquet, FlatGeoBuf ● Supporting infrastructure: duckDB for Geoparquet etc Key limitations & Integration Strategies
  • 81. Lessons Learned ● Cloud native is all about making it easy to publish data without a server, optimizing responses to web data requests: read just what you need! ● No one size fits all: each format has its strengths and limitations ● STAC: steeper learning curve, collections within catalogs and vice versa, implementations vary; security ● COG, COPC: perhaps the most intuitive - 2d and 3d arrays are just easier to manage. STAC/COG most data ● Vector - still evolving: FlatGeoBuf more effective in its cloud native support but newer, less widely adopted. Geoparquet has more tooling but design and support needed to make effectively cloud native ● ZARR - powerful but complex - as a very new and niche format, support is still growing
  • 82. Summary ● Cloud native is all about making it easy to publish data without a server, optimizing responses to web data requests: read just what you need! ● Safe’s strategy is to track and support emerging standards across a range of data types so FME users can stay ahead of evolving web technologies ● FME allows you to integrate between hybrid environments as needed ● Keep the processing close to the data ● Minimize traffic footprint - reader filtering ● Open standards enable community-wide adoption and access ● No one size fits all - know your key requirements & test!
  • 83. One platform, two technologies FME Form FME Flow Build and run data workflows Automate data workflows FME Flow Hosted Safe Software managed instance fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
  • 85. Resources ● Geospatial Cloud Native Overview ● Webinar: Cloud Revolution: Exploring the New Wave of Serverless Spatial Data ● Webinar: Leveraging FME for Cloud Native Databases ● Cloud Native Databases - Blog ● Radiant Earth Blog: Cloud Native Geospatial Solutions ● Cloud Native Geospatial Foundation & Slack channel ● OGC - Cloud Native Geospatial ● Source Cooperative ● Chris Holmes: FOSS4G NA 2023 | Towards a Cloud Native Spatial Data Infrastructure ● guide.cloudnativegeo.org ● Cloud Native Data Formats ● Safe’s Participation in OGC Pilots
  • 86.
  • 87. Data Sources STAC / COG: ● catalogue.dataspace.copernicus.eu/stac/ ● cmr.earthdata.nasa.gov/stac/ ● planetarycomputer.microsoft.com/catalog ● usgs.gov/landsat-missions/landsat-collection-2 ● planetarycomputer.microsoft.com/api/stac/v1/colle ctions/sentinel-2-l2a ● https://planetarycomputer.microsoft.com/api/stac/ v1/collections/nrcan-landcover ZARR: ● https://console.cloud.google.com/marketplace/pro duct/noaa-public/cmip6 COPC: ● github.com/PDAL/data/tree/master/autzen ● copc.io/#example-data
  • 89. Next Steps ● Coming: ○ Blogs ○ Tutorials ● Community involvement: Cloud Native Geospatial Foundation, OGC ● Events: ○ cncf.io/events/ ○ safe.com/company-updates/2024-upcomi ng-events/ ○ ogcmeet.org/ ● New functionality: what are your priorities? ○ DuckDB? ○ ZARR band range? ○ Other STAC media types?
  • 90. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base community.safe.com /s/knowledge-base Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
  • 91. ClaimYour Community Badge ● Get community badges for watching webinars! ● fme.ly/WebinarBadge ● Today’s code: SLMWB Join the Community today!
  • 93. ThankYou Recap of Next Steps 1 Contact us: info@safe.com 2 Check out the new landing page to get started! Please fill out our webinar survey