SlideShare a Scribd company logo
1 of 15
Download to read offline
Scalable Data Analytics and Visualization
with Cloud Optimized Services
Sudhir Raj Shrestha
sshrestha@esri.com
Steve Kopp
skopp@esri.com
AGU Fall Meeting, December 12, 2018 | Washington, DC
Living in Interoperable Open World
ETL
Google Earth
ENVI
Imagine
MapInfo
Ionic
GeoMedia
ArcGIS Server
ArcGIS Desktop
ArcGIS Online
ArcExplorer
Gaia
gvSig
OpenLayers
uDig
QGIS
AutoCAD
Services
Web Services
OGC
ArcGIS Pro
Platform
Open
Open
Specs
Open Science
Open
Data
Open APIOpen
Source
Open
Standards
Open
Platform
Open
Community
Proprietary
Open
ETL
Standards
Interoperable
Compliance
FOSS
Open
Architecture
mosaic dataset
geoprocessing tools
optimization
consumption
multidimensional filter
raster functions
Interactive/dynamic
scalable/extensible
dynamic web services
web maps & apps
web based analytics
Analyze ShareManage VisualizeIngest
raster types
crawl disk
link to pixels
animate over slices
sophisticated renderers
vector & scalar fields
GISworkflows that scale
Designed for the characteristics of Earth Observations scientific data
Data Service
Copyright © Esri. All rights reserved.
Cloud Optimized Formats
• Raw/Striped TIF
- Disadvantage : Sequential access
- Compression:
- None
• Tiled GeoTIF + COG
- Advantage: Tiled Access
- Compression
- None
- Lossy JPEG (8bit & 12bit)
- Lossless Deflate LZW…Generic lossless
• JPEG2000
- Advantage : Tiled Access, Higher Compression
- Disadvantage : Computationally expensive
- Compression
- Lossy
- Lossless
• MRF (Meta Raster Format)
- Advantage : Tiled Access +
Optimization
- Compression
- LERC – Controlled Lossy – Very Fast –
Optimum for higher bit depth and categorical
• CRF (Cloud Raster Format)
Advantage :
Bundles of Tiles, Multi write
- Compression
- LERC
Scalable Cloud based Implementations : High Resolution Rapid Refresh (HRRR)
NWSModel
Generation
NCO Compute
Download
S3/Blob
Upload
Public
Agencies
Create
Mosaic Dataset
Publish
Mosaic Dataset
Elastic
EC2 Instances
NCO/IDP
Internal
AWS|Azure
Auto Scaling group
ArcGIS Portal
Elastic Load
Balancer
EC2
AGS
EC2
Configure Store
Federated with
Portal
S3 Storage User
Raster Store,
zip, fgdb
Elastic IP for Portal
Professional Imagery /
Geospatial Analysts
Client consuming
Imagery Items
Client consuming
Hosted Image
Services
VPC
VPC
Imagery S3 Storage
ArcGIS ProImage Services
Image Server
Image Services
Image Server
EC2 EC2
Image Services
Image Server
EC2
Dynamic Image
Services
Raster Analytics
Elastic Load
Balancer
Client consuming
Dynamic Image
Services
Enterprise + ArcGIS Server + Image server + RDS
• Used when we need to do raster analytics
• Mosaic sit in Postgres ( RDS)
• All server machines are in cluster
• Config store is on a different ec2 machine
More info on how to do this Imageserver Deployment Image server Cloud Formation Template
More info on how to do ArcGIS enterprise Deployment ArcGIS Enterprise Cloud Formation Template
RDS
Postgres RDS
Portal : m5.2xlarge 1 machine
Image server : m5.2xlarge 3 machine
File Store : m5.xlarge 1 machine
Postgres : RDS
Load Balance for Image server stack
Elastic IP for the portal machine
All machine in same VPC
Scalable Cloud Implementation
Global Streamflow Services – A Paradigm Shift
Global
Data
PAST – Individual Hydrologic Forecasting
Global
Services
NOW – Global Hydrologic Forecasting
Current NWS River Fcst
points ~3,600
The National Water Model expands forecasting capabilities
NWM streamflow and velocity for ~
2.7 million river reaches
Current NWS AHPS points overlaid with
NWM Stream Reaches (State level scale)
Courtesy: NOAA OWP
National Water Model Web Service
Real time Flood Inundation Forecast
National Water Model + Height Above Nearest Drainage (HAND)
Compute national Height Above Nearest Drainage (HAND)
4 days of computation on 32 processors
COTS software on standard hardware
NHD Streams
10 meter NED
Elevation
Hourly download
Convert flow to depth
through rating curve
Real time Flood Forecasting
HAND Raster Map
National Water Model
Copyright © Esri. All rights reserved.
Near real time Flood Inundation Forecast
using HAND (NOAA National Water Model)
Copyright © Esri. All rights reserved.
Dynamic scientific data computing processing:
Web Map Application: HRRR
Copyright © Esri. All rights reserved.
Space-Time Ocean Data:
Web Map Application: HYCOM

More Related Content

What's hot

01 supermapiportaloverview
01 supermapiportaloverview01 supermapiportaloverview
01 supermapiportaloverviewGeoMedeelel
 
AWS Customer Presentation - Cloud Made
AWS Customer Presentation - Cloud MadeAWS Customer Presentation - Cloud Made
AWS Customer Presentation - Cloud MadeAmazon Web Services
 
02 supermapiclientforjavascriptintroduction
02 supermapiclientforjavascriptintroduction02 supermapiclientforjavascriptintroduction
02 supermapiclientforjavascriptintroductionGeoMedeelel
 
The IoT and big data
The IoT and big dataThe IoT and big data
The IoT and big dataGal Ben-Haim
 
Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...
Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...
Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...Data Con LA
 
Cartaro - Geospatial CMS (en)
Cartaro - Geospatial CMS (en)Cartaro - Geospatial CMS (en)
Cartaro - Geospatial CMS (en)Uli Müller
 
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...Maya Lumbroso
 
SnapLogic Live: AWS Integration
SnapLogic Live: AWS IntegrationSnapLogic Live: AWS Integration
SnapLogic Live: AWS IntegrationSnapLogic
 
Large Scale Geo Processing on Hadoop
Large Scale Geo Processing on HadoopLarge Scale Geo Processing on Hadoop
Large Scale Geo Processing on HadoopChristoph Körner
 
Designing and Using Cached Map
Designing and Using Cached Map Designing and Using Cached Map
Designing and Using Cached Map M.Muneeb Ashraf
 
2013 Geospatial Data and Project Management Track, Mobile Big Data by Genie Hays
2013 Geospatial Data and Project Management Track, Mobile Big Data by Genie Hays2013 Geospatial Data and Project Management Track, Mobile Big Data by Genie Hays
2013 Geospatial Data and Project Management Track, Mobile Big Data by Genie HaysGIS in the Rockies
 
JanusGraph: Looking Backward, Reaching Forward
JanusGraph: Looking Backward, Reaching ForwardJanusGraph: Looking Backward, Reaching Forward
JanusGraph: Looking Backward, Reaching ForwardJason Plurad
 
Overkill Analytics Seattle Spark Meetup
Overkill Analytics Seattle Spark MeetupOverkill Analytics Seattle Spark Meetup
Overkill Analytics Seattle Spark MeetupClaudiu Barbura
 
Computation of spatial data on Hadoop Cluster
Computation of spatial data on Hadoop ClusterComputation of spatial data on Hadoop Cluster
Computation of spatial data on Hadoop ClusterAbhishek Sagar
 
Working with arcgis online
Working with arcgis onlineWorking with arcgis online
Working with arcgis onlinezmmin383
 
Making the most of raster data from the arcgis living atlas of the world
Making the most of raster data from the arcgis living atlas of the worldMaking the most of raster data from the arcgis living atlas of the world
Making the most of raster data from the arcgis living atlas of the worldAileen Buckley
 
Making the most of raster data from the arcgis living atlas of the world
Making the most of raster data from the arcgis living atlas of the worldMaking the most of raster data from the arcgis living atlas of the world
Making the most of raster data from the arcgis living atlas of the worldAileen Buckley
 

What's hot (20)

01 supermapiportaloverview
01 supermapiportaloverview01 supermapiportaloverview
01 supermapiportaloverview
 
AWS Customer Presentation - Cloud Made
AWS Customer Presentation - Cloud MadeAWS Customer Presentation - Cloud Made
AWS Customer Presentation - Cloud Made
 
02 supermapiclientforjavascriptintroduction
02 supermapiclientforjavascriptintroduction02 supermapiclientforjavascriptintroduction
02 supermapiclientforjavascriptintroduction
 
The IoT and big data
The IoT and big dataThe IoT and big data
The IoT and big data
 
AWS Customer Presentation- Esri
AWS Customer Presentation- EsriAWS Customer Presentation- Esri
AWS Customer Presentation- Esri
 
Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...
Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...
Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...
 
Cartaro - Geospatial CMS (en)
Cartaro - Geospatial CMS (en)Cartaro - Geospatial CMS (en)
Cartaro - Geospatial CMS (en)
 
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...
 
SnapLogic Live: AWS Integration
SnapLogic Live: AWS IntegrationSnapLogic Live: AWS Integration
SnapLogic Live: AWS Integration
 
Large Scale Geo Processing on Hadoop
Large Scale Geo Processing on HadoopLarge Scale Geo Processing on Hadoop
Large Scale Geo Processing on Hadoop
 
Designing and Using Cached Map
Designing and Using Cached Map Designing and Using Cached Map
Designing and Using Cached Map
 
2013 Geospatial Data and Project Management Track, Mobile Big Data by Genie Hays
2013 Geospatial Data and Project Management Track, Mobile Big Data by Genie Hays2013 Geospatial Data and Project Management Track, Mobile Big Data by Genie Hays
2013 Geospatial Data and Project Management Track, Mobile Big Data by Genie Hays
 
Cloud computer
Cloud computerCloud computer
Cloud computer
 
JanusGraph: Looking Backward, Reaching Forward
JanusGraph: Looking Backward, Reaching ForwardJanusGraph: Looking Backward, Reaching Forward
JanusGraph: Looking Backward, Reaching Forward
 
Overkill Analytics Seattle Spark Meetup
Overkill Analytics Seattle Spark MeetupOverkill Analytics Seattle Spark Meetup
Overkill Analytics Seattle Spark Meetup
 
Computation of spatial data on Hadoop Cluster
Computation of spatial data on Hadoop ClusterComputation of spatial data on Hadoop Cluster
Computation of spatial data on Hadoop Cluster
 
Working with arcgis online
Working with arcgis onlineWorking with arcgis online
Working with arcgis online
 
Making the most of raster data from the arcgis living atlas of the world
Making the most of raster data from the arcgis living atlas of the worldMaking the most of raster data from the arcgis living atlas of the world
Making the most of raster data from the arcgis living atlas of the world
 
Making the most of raster data from the arcgis living atlas of the world
Making the most of raster data from the arcgis living atlas of the worldMaking the most of raster data from the arcgis living atlas of the world
Making the most of raster data from the arcgis living atlas of the world
 
QGIS - Free alternative to ArcMap
QGIS - Free alternative to ArcMapQGIS - Free alternative to ArcMap
QGIS - Free alternative to ArcMap
 

Similar to Scalable Data Analytics and Visualization with Cloud Optimized Services

Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...Data Con LA
 
Architecting the ArcGIS Platform
Architecting the ArcGIS PlatformArchitecting the ArcGIS Platform
Architecting the ArcGIS PlatformEsri UK
 
Welsh Conference 2023 Opening Plenary
Welsh Conference 2023 Opening PlenaryWelsh Conference 2023 Opening Plenary
Welsh Conference 2023 Opening PlenaryEsri UK
 
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and AnalyticsDataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and AnalyticsDataStax Academy
 
Shark SQL and Rich Analytics at Scale
Shark SQL and Rich Analytics at ScaleShark SQL and Rich Analytics at Scale
Shark SQL and Rich Analytics at ScaleDataWorks Summit
 
ArcGIS - A Platform for Developers & Startups
ArcGIS - A Platform for Developers & StartupsArcGIS - A Platform for Developers & Startups
ArcGIS - A Platform for Developers & StartupsEsri Ireland
 
Imagery and beyond - BK 2016
Imagery and beyond - BK 2016Imagery and beyond - BK 2016
Imagery and beyond - BK 2016Geodata AS
 
NDGeospatialSummit2019 - ArcGIS Enterprise – Power Your Location Intelligence
NDGeospatialSummit2019 - ArcGIS Enterprise – Power Your Location IntelligenceNDGeospatialSummit2019 - ArcGIS Enterprise – Power Your Location Intelligence
NDGeospatialSummit2019 - ArcGIS Enterprise – Power Your Location IntelligenceNorth Dakota GIS Hub
 
GIS as a Platform by Sam Viana (Esri Inc)
GIS as a Platform by Sam Viana (Esri Inc)GIS as a Platform by Sam Viana (Esri Inc)
GIS as a Platform by Sam Viana (Esri Inc)Esri South Africa
 
Mel McIntyre, OpenApp.ie LGMA
Mel McIntyre, OpenApp.ie LGMAMel McIntyre, OpenApp.ie LGMA
Mel McIntyre, OpenApp.ie LGMAOpenSourceLGMA
 
Nvidia gpu-application-catalog TESLA K80 GPU應用程式型錄
Nvidia gpu-application-catalog TESLA K80 GPU應用程式型錄Nvidia gpu-application-catalog TESLA K80 GPU應用程式型錄
Nvidia gpu-application-catalog TESLA K80 GPU應用程式型錄Cheer Chain Enterprise Co., Ltd.
 
Dsm Presentation
Dsm PresentationDsm Presentation
Dsm Presentationrichoe
 
Open Source GIS
Open Source GISOpen Source GIS
Open Source GISJoe Larson
 
Esri South Africa ArcGIS 10.2 Rollout
Esri South Africa ArcGIS 10.2 Rollout Esri South Africa ArcGIS 10.2 Rollout
Esri South Africa ArcGIS 10.2 Rollout Esri South Africa
 
Strategies for Integrating Utility System Operational Data into ArcGIS Server...
Strategies for Integrating Utility System Operational Data into ArcGIS Server...Strategies for Integrating Utility System Operational Data into ArcGIS Server...
Strategies for Integrating Utility System Operational Data into ArcGIS Server...True North Geographic Technologies
 
Modernizing upstream workflows with aws storage - john mallory
Modernizing upstream workflows with aws storage -  john malloryModernizing upstream workflows with aws storage -  john mallory
Modernizing upstream workflows with aws storage - john malloryAmazon Web Services
 

Similar to Scalable Data Analytics and Visualization with Cloud Optimized Services (20)

Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
 
Architecting the ArcGIS Platform
Architecting the ArcGIS PlatformArchitecting the ArcGIS Platform
Architecting the ArcGIS Platform
 
Welsh Conference 2023 Opening Plenary
Welsh Conference 2023 Opening PlenaryWelsh Conference 2023 Opening Plenary
Welsh Conference 2023 Opening Plenary
 
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and AnalyticsDataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
 
Shark SQL and Rich Analytics at Scale
Shark SQL and Rich Analytics at ScaleShark SQL and Rich Analytics at Scale
Shark SQL and Rich Analytics at Scale
 
ArcGIS - A Platform for Developers & Startups
ArcGIS - A Platform for Developers & StartupsArcGIS - A Platform for Developers & Startups
ArcGIS - A Platform for Developers & Startups
 
Imagery and beyond - BK 2016
Imagery and beyond - BK 2016Imagery and beyond - BK 2016
Imagery and beyond - BK 2016
 
Leveraging GIS with AutoCAD
Leveraging GIS with AutoCADLeveraging GIS with AutoCAD
Leveraging GIS with AutoCAD
 
ArcGIS and Multi-D: Tools & Roadmap
ArcGIS and Multi-D: Tools & RoadmapArcGIS and Multi-D: Tools & Roadmap
ArcGIS and Multi-D: Tools & Roadmap
 
NDGeospatialSummit2019 - ArcGIS Enterprise – Power Your Location Intelligence
NDGeospatialSummit2019 - ArcGIS Enterprise – Power Your Location IntelligenceNDGeospatialSummit2019 - ArcGIS Enterprise – Power Your Location Intelligence
NDGeospatialSummit2019 - ArcGIS Enterprise – Power Your Location Intelligence
 
GIS as a Platform by Sam Viana (Esri Inc)
GIS as a Platform by Sam Viana (Esri Inc)GIS as a Platform by Sam Viana (Esri Inc)
GIS as a Platform by Sam Viana (Esri Inc)
 
Mel McIntyre, OpenApp.ie LGMA
Mel McIntyre, OpenApp.ie LGMAMel McIntyre, OpenApp.ie LGMA
Mel McIntyre, OpenApp.ie LGMA
 
Nvidia gpu-application-catalog TESLA K80 GPU應用程式型錄
Nvidia gpu-application-catalog TESLA K80 GPU應用程式型錄Nvidia gpu-application-catalog TESLA K80 GPU應用程式型錄
Nvidia gpu-application-catalog TESLA K80 GPU應用程式型錄
 
Dsm Presentation
Dsm PresentationDsm Presentation
Dsm Presentation
 
Open Source GIS
Open Source GISOpen Source GIS
Open Source GIS
 
Esri South Africa ArcGIS 10.2 Rollout
Esri South Africa ArcGIS 10.2 Rollout Esri South Africa ArcGIS 10.2 Rollout
Esri South Africa ArcGIS 10.2 Rollout
 
Strategies for Integrating Utility System Operational Data into ArcGIS Server...
Strategies for Integrating Utility System Operational Data into ArcGIS Server...Strategies for Integrating Utility System Operational Data into ArcGIS Server...
Strategies for Integrating Utility System Operational Data into ArcGIS Server...
 
Multidimensional Scientific Data in ArcGIS
Multidimensional Scientific Data in ArcGISMultidimensional Scientific Data in ArcGIS
Multidimensional Scientific Data in ArcGIS
 
Advancing Scientific Data Support in ArcGIS
Advancing Scientific Data Support in ArcGISAdvancing Scientific Data Support in ArcGIS
Advancing Scientific Data Support in ArcGIS
 
Modernizing upstream workflows with aws storage - john mallory
Modernizing upstream workflows with aws storage -  john malloryModernizing upstream workflows with aws storage -  john mallory
Modernizing upstream workflows with aws storage - john mallory
 

More from Globus

The Department of Energy's Integrated Research Infrastructure (IRI).pdf
The Department of Energy's Integrated Research Infrastructure (IRI).pdfThe Department of Energy's Integrated Research Infrastructure (IRI).pdf
The Department of Energy's Integrated Research Infrastructure (IRI).pdfGlobus
 
Research Automation with Globus Flows.pptx
Research Automation with Globus Flows.pptxResearch Automation with Globus Flows.pptx
Research Automation with Globus Flows.pptxGlobus
 
Reactive Documents and Computational Pipelines
Reactive Documents and Computational PipelinesReactive Documents and Computational Pipelines
Reactive Documents and Computational PipelinesGlobus
 
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisProviding Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisGlobus
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageGlobus
 
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...Globus
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
 
GlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote AddressGlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote AddressGlobus
 
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use CasesGlobus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use CasesGlobus
 
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) WorkflowsGlobus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) WorkflowsGlobus
 
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...Globus
 
Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)Globus
 
Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)Globus
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsGlobus
 
Extending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureExtending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureGlobus
 
Enhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptxEnhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptxGlobus
 
Enhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdfEnhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdfGlobus
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
 
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...Globus
 
Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024Globus
 

More from Globus (20)

The Department of Energy's Integrated Research Infrastructure (IRI).pdf
The Department of Energy's Integrated Research Infrastructure (IRI).pdfThe Department of Energy's Integrated Research Infrastructure (IRI).pdf
The Department of Energy's Integrated Research Infrastructure (IRI).pdf
 
Research Automation with Globus Flows.pptx
Research Automation with Globus Flows.pptxResearch Automation with Globus Flows.pptx
Research Automation with Globus Flows.pptx
 
Reactive Documents and Computational Pipelines
Reactive Documents and Computational PipelinesReactive Documents and Computational Pipelines
Reactive Documents and Computational Pipelines
 
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisProviding Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
GlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote AddressGlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote Address
 
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use CasesGlobus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
 
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) WorkflowsGlobus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
 
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
 
Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)
 
Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Extending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureExtending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data Infrastructure
 
Enhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptxEnhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptx
 
Enhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdfEnhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdf
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
 
Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024
 

Recently uploaded

TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024Stephen Perrenod
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityScyllaDB
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfUK Journal
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Julian Hyde
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxJennifer Lim
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfFIDO Alliance
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceSamy Fodil
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...FIDO Alliance
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutesconfluent
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPTiSEO AI
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessUXDXConf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireExakis Nelite
 

Recently uploaded (20)

TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 

Scalable Data Analytics and Visualization with Cloud Optimized Services

  • 1. Scalable Data Analytics and Visualization with Cloud Optimized Services Sudhir Raj Shrestha sshrestha@esri.com Steve Kopp skopp@esri.com AGU Fall Meeting, December 12, 2018 | Washington, DC
  • 2. Living in Interoperable Open World ETL Google Earth ENVI Imagine MapInfo Ionic GeoMedia ArcGIS Server ArcGIS Desktop ArcGIS Online ArcExplorer Gaia gvSig OpenLayers uDig QGIS AutoCAD Services Web Services OGC ArcGIS Pro
  • 4. mosaic dataset geoprocessing tools optimization consumption multidimensional filter raster functions Interactive/dynamic scalable/extensible dynamic web services web maps & apps web based analytics Analyze ShareManage VisualizeIngest raster types crawl disk link to pixels animate over slices sophisticated renderers vector & scalar fields GISworkflows that scale Designed for the characteristics of Earth Observations scientific data Data Service
  • 5. Copyright © Esri. All rights reserved. Cloud Optimized Formats • Raw/Striped TIF - Disadvantage : Sequential access - Compression: - None • Tiled GeoTIF + COG - Advantage: Tiled Access - Compression - None - Lossy JPEG (8bit & 12bit) - Lossless Deflate LZW…Generic lossless • JPEG2000 - Advantage : Tiled Access, Higher Compression - Disadvantage : Computationally expensive - Compression - Lossy - Lossless • MRF (Meta Raster Format) - Advantage : Tiled Access + Optimization - Compression - LERC – Controlled Lossy – Very Fast – Optimum for higher bit depth and categorical • CRF (Cloud Raster Format) Advantage : Bundles of Tiles, Multi write - Compression - LERC
  • 6. Scalable Cloud based Implementations : High Resolution Rapid Refresh (HRRR) NWSModel Generation NCO Compute Download S3/Blob Upload Public Agencies Create Mosaic Dataset Publish Mosaic Dataset Elastic EC2 Instances NCO/IDP Internal AWS|Azure
  • 7. Auto Scaling group ArcGIS Portal Elastic Load Balancer EC2 AGS EC2 Configure Store Federated with Portal S3 Storage User Raster Store, zip, fgdb Elastic IP for Portal Professional Imagery / Geospatial Analysts Client consuming Imagery Items Client consuming Hosted Image Services VPC VPC Imagery S3 Storage ArcGIS ProImage Services Image Server Image Services Image Server EC2 EC2 Image Services Image Server EC2 Dynamic Image Services Raster Analytics Elastic Load Balancer Client consuming Dynamic Image Services Enterprise + ArcGIS Server + Image server + RDS • Used when we need to do raster analytics • Mosaic sit in Postgres ( RDS) • All server machines are in cluster • Config store is on a different ec2 machine More info on how to do this Imageserver Deployment Image server Cloud Formation Template More info on how to do ArcGIS enterprise Deployment ArcGIS Enterprise Cloud Formation Template RDS Postgres RDS Portal : m5.2xlarge 1 machine Image server : m5.2xlarge 3 machine File Store : m5.xlarge 1 machine Postgres : RDS Load Balance for Image server stack Elastic IP for the portal machine All machine in same VPC Scalable Cloud Implementation
  • 8. Global Streamflow Services – A Paradigm Shift Global Data PAST – Individual Hydrologic Forecasting Global Services NOW – Global Hydrologic Forecasting
  • 9. Current NWS River Fcst points ~3,600 The National Water Model expands forecasting capabilities NWM streamflow and velocity for ~ 2.7 million river reaches Current NWS AHPS points overlaid with NWM Stream Reaches (State level scale) Courtesy: NOAA OWP
  • 10. National Water Model Web Service
  • 11. Real time Flood Inundation Forecast National Water Model + Height Above Nearest Drainage (HAND) Compute national Height Above Nearest Drainage (HAND) 4 days of computation on 32 processors COTS software on standard hardware NHD Streams 10 meter NED Elevation Hourly download Convert flow to depth through rating curve Real time Flood Forecasting HAND Raster Map National Water Model
  • 12.
  • 13. Copyright © Esri. All rights reserved. Near real time Flood Inundation Forecast using HAND (NOAA National Water Model)
  • 14. Copyright © Esri. All rights reserved. Dynamic scientific data computing processing: Web Map Application: HRRR
  • 15. Copyright © Esri. All rights reserved. Space-Time Ocean Data: Web Map Application: HYCOM