This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) and netCDF (network Common Data Form) are two widely used data formats for storing and manipulating scientific data. The NetCDF format also supports temporal data by using multidimensional arrays. The basic structure of data in this format and how to work with it will be covered in the context of standardized data structures and conventions. This slide will demonstrate the tools and techniques for ingesting HDF and netCDF data efficiently in ArcGIS, as well as some common workflows to employ the visualization capabilities of ArcGIS for effective animation and analysis of your data.

1. Working with HDF and netCDF Data in ArcGIS
Tools and Case Studies
Nawajish Noman
Jeff Donze
HDF and HDF-EOS Workshop XIV, September 28-30, 2010, Champaign, IL
HDF28-

2. Outline
• HDF and netCDF in ArcGIS
• Time in ArcGIS
• Performing Analysis
• Use Cases and Applications
• Script Tools
•F t
Future Directions
Di
ti

3. HDF and netCDF in ArcGIS

4. Scientific Data and ESRI
• Direct support - NetCDF and HDF
• THREDDS – a data server technology for multidimensional array
data, integrated use by our customers
• Examples using ESRI technology
• National Climate Data Center
• National Weather Service
• National Center for
Atmospheric Research
• Air Force Weather
• Australian Navy
• Australian Bur.of Met.
• UK Met Office

5. HDF4 and HDF5 Support in ArcGIS

6. Displaying MODIS LST Data
HDFView
ArcGIS

7. HDF4 and HDF5 Support in ArcGIS
http://help.arcgis.com/en/arcgisdesktop/10.0/help/

8. What is NetCDF?
NetCDF?
• NetCDF (network Common Data Form)
network
A platform independent format for representing multi-dimensional
multiarrayarray-orientated scientific data.
• Self Describing: a netCDF file includes information about the data it
contains.
• Di
Direct A
t Access: a small subset of a large dataset may be accessed
ll b t f l
d t
t
b
d
efficiently, without first reading through all the preceding data.
• Sh
Sharable: one writer and multiple readers may simultaneously
bl
it
d
lti l
d
i lt
l
access the same netCDF file.
NetCDF is relatively new to the GIS community but widely used by scientific
communities for many years.

9. Storing Data in a netCDF File
Time = 1 to 3
Y = 1 to 4
X = 1 to 4

10. NetCDF Support in ArcGIS
• ArcGIS reads/writes netCDF since version 9.2
• An array based data structure for storing
multidimensional data.
T
• N-dimensional coordinates systems
• X, Y, Z, time, and other dimensions
• Variables – support for multiple variables
• Temperature, humidity, pressure, salinity, etc
Temperature,
• Geometry – implicit or explicit
• Regular grid (implicit)
(implicit)
• Irregular grid
• Points
Y
Z
X

11. Gridded Data
Regular Grid
Irregular Grid

12. Ingesting netCDF data in ArcGIS
• NetCDF data is accessed as
Raster
• Feature
• Table
•
• Di
Direct read
d
• Exports GIS data to netCDF

13. CF Convention
Climate and Forecast (CF) Convention
(CF)
http://cf-pcmdi.llnl.gov/
htt // f
di ll l
/
Initially developed for
• Climate and forecast data
• Atmosphere, surface and ocean model-generated data
model• Also for observational datasets
• The CF conventions generalize and extend the COARDS (Cooperative
cean/A
Ocean/Atmosphere Research Data Service) convention.
• CF i now th most widely used conventions f geospatial netCDF
is
the
t id l
d
ti
for
ti l tCDF
data. It has the best coordinate system handling.

14. NetCDF and Coordinate Systems
• Geographic Coordinate Systems (GCS)
X dimension units: degrees_east
• Y dimension units: degrees_north
•
• Projected Coordinate Systems (PCS)
X dimension standard_name: projection_x_coordinate
standard_name:
• Y dimension standard_name: projection_y_coordinate
standard_name:
• V i bl has a grid_mapping attribute.
Variable h
id
i
tt ib t
• CF 1.4 conventions currently supports twelve predefined coordinate
systems (Appendix F: Grid Mappings)
Mappings)
•
• Undefined
• If not GCS or PCS
• ArcGIS writes (and recognizes) PE String as a variable attribute.

15. NetCDF Tools
Toolbox: Multidimension Tools
Make NetCDF Raster Layer
• Make NetCDF Feature Layer
• Make NetCDF Table View
•
Raster to NetCDF
• Feature to NetCDF
• Table to NetCDF
•
•
Select by Dimension
y

16. NetCDF Layer/Table Properties
Raster
Feature
Feat re
Table

17. Changing Time Slice
Time = 1

18. Using NetCDF Data
Behaves the same as any layer or table
• Display
• Same display tools for raster and feature layers will work on netCDF
raster and netCDF feature layers.
• Graphing
• Driven by the table just like any other chart.
• Animation
• Multidimensional data can be animated through a dimension (e.g.
time, pressure, elevation)
• Analysis Tools
• A netCDF layer or table will work just like any other raster layer,
feature layer or table (e g create buffers around netCDF points
layer,
table. (e.g.
points,
reproject rasters, query tables, etc.)
rasters,

19. Help on netCDF

20. Time in ArcGIS

21. Why visualize data through time?

22. Temporal GIS Patterns
Dynamic
Discrete
Stationary
Change
something that moves
something that
“just happens”
“j t h
”
stands still but
records changes
change or growth
• Planes
• Vehicles
• Animals
• Satellites
• Storms
• Crimes
• Weather Stations
• Population
• Lightning
• Accidents
• Traffic Sensors
• Distribution
• Fire Perimeter

23. GIS Integration of Time
New Ways to Manage Visualize & Analyze Geography
Manage,
Multi
Dimensional
Data (netCDF)
(netCDF)
Visualize Change
Vi
li Ch
T
Real Time
Sensor Network
y
x
Files
DBMS
Mobile
Stationary
T1
Simulation
Modeling

24. Time is now built-in to ArcGIS
built-
•
Simple Temporal Mapping
•
Unified experience for Time
•
•
•
Configure time properties on the layer
Use Time Slider to visualize temporal data
is ali e
Share temporal visualization
TimeTime-enabled Map Services
Export videos or images
Generate temporal map books
using ArcPy scripting
• L
Layer and map packages
d
k
•
•
•

25. Visualizing temporal data using graphs
• Create a graph using a layer or table
• Create an animation in the usual way, attaching the
layer or table to a time layer track
• When the animation is played,
p y ,
the graph will animate

26. Animation examples
1979

27. Performing Analysis

28. Spatial and Temporal Analysis
• Several hundreds analytical tools available for raster, features,
and table
• Temporal Modeling
p
g
• Looping and iteration in ModelBuilder and Python

29. Generate Rainfall Statistics
• Calculates specified statistics for all time steps
• Outputs a raster catalog
• Optionally outputs a netCDF file

30. Generate Rainfall Statistics Table
• Calculates statistics for all time steps
• Outputs a table
• Optionally creates a graph

31. Demo

32. Use Cases and Applications
•
•
•
•
•
Hydrography and METOC Branch, Royal Australian Navy
Applied Science Associated, Inc.
Inc.
The Nature Conservancy
The University of Washington
The University of Southern Mississippi

33. Observations and Model Forecasts
• Near real-time observations
real•
•
•
•
•
Weather satellite imagery
Sea Surface Temperature
Significant Wave Height (altimeter)
Ocean Winds
Moisture and Precipitation
• Numerical model forecasts
• Fixed domain Global, Regional, Tropical and Local atmospheric models
(ACCESS)
• Fixed domain Global, Regional, and Local wave models (WaveWatch 3)
(WaveWatch
• Global Ocean Model (BLUElink)
(BLUElink)
highg
p
• Relocatable high-resolution atmospheric and ocean models
Source: Hydrography and METOC Branch, Royal Australian Navy.

34. Climatology – CCMP Winds
Source: Hydrography and METOC Branch, Royal Australian Navy.

35. Climatology – CCMP Winds
Source: Hydrography and METOC Branch, Royal Australian Navy.

36. Ocean Model Forecast – Custom Display
Source: Hydrography and METOC Branch, Royal Australian Navy.

37. Web Services - GHRSST
Source: Hydrography and METOC Branch, Royal Australian Navy.

38. Environmental Data Connector (EDC)
XML
DAP
THREDDS
Features
NetCDF
Java App,
Unidata
Rasters
ArcGIS
Extension
Source: Applied Science Associated, Inc.

39. EDC Screen and Time Slider Toolbar
Synchronize and animate time-varying data in
http://www.pfeg.noaa.gov/products/EDC/
http://www.asascience.com/TimeSlider/index.htm
Source: Applied Science Associated, Inc.

40. Time Series Graphs
Source: Applied Science Associated, Inc.

41. Climate Wizard
http://www.climatewizard.org/
Developers: The Nature Conservancy, The University of Washington, and The University of Southern Mississippi

42. Climate Change
http://tncclimate.esri.com/TNC/ClimateChange.html
Developers: The Nature Conservancy, The University of Washington, and ESRI

43. Script Tools

44. Community Developed Tools
• Geoprocessing Resource Center
http://resources.arcgis.com/geoprocessing/
• Marine Geospatial
Ecology Tools
• Developed at Duke Univ.
• Over 180 tools for import
management, and
analysis of marine data
• Australian Navy tools
(not publicly available)

45. Script Tools of interest
• Python is used to build custom tools for specific tasks or
datasets

46. NEXRAD Geoprocessing Tools
•
•
•
•
•
•
Currently 6 geoprocessing script tools
Designed to work with NEXRAD netCDF file
Can be easily modified for other datasets
Customized tools for various workflows
Simplify repetitive work
Automate GIS processes

47. Download NEXRAD NetCDF File
• Downloads netCDF file from the THREDDS server

48. Extract NEXRAD Rainfall To Points
• Extracts the cell values for all time steps
• Outputs a feature class

49. NEXRAD NetCDF To Raster Catalog
• Creates a raster catalog
• Each catalog item corresponds to a time slice in the netCDF file

50. Future Directions

51. Things to Consider…
• Embrace the Common Data Model (netCDF, HDF etc.)
(netCDF,
• Use Data and metadata standards (OGC, CF etc)
• Make your data “spatial” (by specifying geographic or a
p j
projected coordinate system)
y
)
• Clearly define workflow and requirements
• Create sample tools where possible

52. Tools Under Consideration
• Temporal Statistics
• Get Variable Statistics
Analysis
•
•
•
•
Clip
p
Extract By Variable
Extract By Dimension
Append By Dimension
Data Management

53. Future directions
• Multidimension data management
• Temporal analysis tools
• Additional support for HDF5 using netCDF 4.x library
• What do you need?

54. Questions?
Nawajish Noman
Lead Product Engineer
nnoman@esri.com
@ i
Jeff Donze
ESRI Federal Business Development
F d lB i
D
l
t
jdonze@esri.com

55. Sharing Temporal Maps & Data with ArcGIS 10
•
Publish time-aware maps
time-
•
Export videos or images, layer and map packages
•
Vi
li data
Visualize d t
Access via REST API
Web API
• FLEX
• JavaScript
• Silverlight
• Time Slider web control
•
•
1979

56. How to store temporal data?
• DATE is a special field type specific to time
• GeoDatabase provides DATE
• If at all possible – use DATE type
• DATE field should be indexed for faster query performance
• Numeric and String fields
•
•
•
•
YYYY
YYYYMM
YYYYMMDD
YYYYMMDDhhmmss
YYYY/MM/DD
YYYY/MM/DD hh:mm:ss
• Time dimension in a netCDF variable
YYYY-MM-DD
YYYY-MMYYYY-MM-DD hh:mm:ss
YYYY-MM-

57. Animation examples
Data source: Cavalieri, D., C. Parkinson, P. Gloerson,
and H.J. Zwally. 1996, updated 2005. Sea ice
concentrations from Nimbus-7 SMMR and DMSP SSM/I
Nimbuspassive microwave data, June to September 2001.
Boulder, CO, USA: National Snow and Ice Data Center.
Sea ice concentration
Solar Insolation
Data provided courtesy of Declan Butler - http://declanbutler.info/blog/
Stream flow analysis
Avian Influenza

58. Atmospheric Model Forecast – Surface Wind
Source: Hydrography and METOC Branch, Royal Australian Navy.