This document discusses scalable data analytics and visualization using cloud optimized services. It provides examples of leveraging cloud technologies like AWS and Azure to build scalable implementations for processing and serving large geospatial and earth observation datasets. This includes architectures for hosting high resolution raster data and services, global hydrologic modeling, and near real-time flood forecasting using NOAA's National Water Model.

1. Scalable Data Analytics and Visualization
with Cloud Optimized Services
Sudhir Raj Shrestha
sshrestha@esri.com
Steve Kopp
skopp@esri.com
AGU Fall Meeting, December 12, 2018 | Washington, DC

2. Living in Interoperable Open World
ETL
Google Earth
ENVI
Imagine
MapInfo
Ionic
GeoMedia
ArcGIS Server
ArcGIS Desktop
ArcGIS Online
ArcExplorer
Gaia
gvSig
OpenLayers
uDig
QGIS
AutoCAD
Services
Web Services
OGC
ArcGIS Pro

3. Platform
Open
Open
Specs
Open Science
Open
Data
Open APIOpen
Source
Open
Standards
Open
Platform
Open
Community
Proprietary
Open
ETL
Standards
Interoperable
Compliance
FOSS
Open
Architecture

4. mosaic dataset
geoprocessing tools
optimization
consumption
multidimensional filter
raster functions
Interactive/dynamic
scalable/extensible
dynamic web services
web maps & apps
web based analytics
Analyze ShareManage VisualizeIngest
raster types
crawl disk
link to pixels
animate over slices
sophisticated renderers
vector & scalar fields
GISworkflows that scale
Designed for the characteristics of Earth Observations scientific data
Data Service

5. Copyright © Esri. All rights reserved.
Cloud Optimized Formats
• Raw/Striped TIF
- Disadvantage : Sequential access
- Compression:
- None
• Tiled GeoTIF + COG
- Advantage: Tiled Access
- Compression
- None
- Lossy JPEG (8bit & 12bit)
- Lossless Deflate LZW…Generic lossless
• JPEG2000
- Advantage : Tiled Access, Higher Compression
- Disadvantage : Computationally expensive
- Compression
- Lossy
- Lossless
• MRF (Meta Raster Format)
- Advantage : Tiled Access +
Optimization
- Compression
- LERC – Controlled Lossy – Very Fast –
Optimum for higher bit depth and categorical
• CRF (Cloud Raster Format)
Advantage :
Bundles of Tiles, Multi write
- Compression
- LERC

6. Scalable Cloud based Implementations : High Resolution Rapid Refresh (HRRR)
NWSModel
Generation
NCO Compute
Download
S3/Blob
Upload
Public
Agencies
Create
Mosaic Dataset
Publish
Mosaic Dataset
Elastic
EC2 Instances
NCO/IDP
Internal
AWS|Azure

7. Auto Scaling group
ArcGIS Portal
Elastic Load
Balancer
EC2
AGS
EC2
Configure Store
Federated with
Portal
S3 Storage User
Raster Store,
zip, fgdb
Elastic IP for Portal
Professional Imagery /
Geospatial Analysts
Client consuming
Imagery Items
Client consuming
Hosted Image
Services
VPC
VPC
Imagery S3 Storage
ArcGIS ProImage Services
Image Server
Image Services
Image Server
EC2 EC2
Image Services
Image Server
EC2
Dynamic Image
Services
Raster Analytics
Elastic Load
Balancer
Client consuming
Dynamic Image
Services
Enterprise + ArcGIS Server + Image server + RDS
• Used when we need to do raster analytics
• Mosaic sit in Postgres ( RDS)
• All server machines are in cluster
• Config store is on a different ec2 machine
More info on how to do this Imageserver Deployment Image server Cloud Formation Template
More info on how to do ArcGIS enterprise Deployment ArcGIS Enterprise Cloud Formation Template
RDS
Postgres RDS
Portal : m5.2xlarge 1 machine
Image server : m5.2xlarge 3 machine
File Store : m5.xlarge 1 machine
Postgres : RDS
Load Balance for Image server stack
Elastic IP for the portal machine
All machine in same VPC
Scalable Cloud Implementation

8. Global Streamflow Services – A Paradigm Shift
Global
Data
PAST – Individual Hydrologic Forecasting
Global
Services
NOW – Global Hydrologic Forecasting

9. Current NWS River Fcst
points ~3,600
The National Water Model expands forecasting capabilities
NWM streamflow and velocity for ~
2.7 million river reaches
Current NWS AHPS points overlaid with
NWM Stream Reaches (State level scale)
Courtesy: NOAA OWP

10. National Water Model Web Service

11. Real time Flood Inundation Forecast
National Water Model + Height Above Nearest Drainage (HAND)
Compute national Height Above Nearest Drainage (HAND)
4 days of computation on 32 processors
COTS software on standard hardware
NHD Streams
10 meter NED
Elevation
Hourly download
Convert flow to depth
through rating curve
Real time Flood Forecasting
HAND Raster Map
National Water Model

13. Copyright © Esri. All rights reserved.
Near real time Flood Inundation Forecast
using HAND (NOAA National Water Model)

14. Copyright © Esri. All rights reserved.
Dynamic scientific data computing processing:
Web Map Application: HRRR

15. Copyright © Esri. All rights reserved.
Space-Time Ocean Data:
Web Map Application: HYCOM