Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies


Published on

This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) and netCDF (network Common Data Form) are two widely used data formats for storing and manipulating scientific data. The NetCDF format also supports temporal data by using multidimensional arrays. The basic structure of data in this format and how to work with it will be covered in the context of standardized data structures and conventions. This slide will demonstrate the tools and techniques for ingesting HDF and netCDF data efficiently in ArcGIS, as well as some common workflows to employ the visualization capabilities of ArcGIS for effective animation and analysis of your data.

Published in: Technology

Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies

  1. 1. Working with HDF and netCDF Data in ArcGIS Tools and Case Studies Nawajish Noman Jeff Donze HDF and HDF-EOS Workshop XIV, September 28-30, 2010, Champaign, IL HDF28-
  2. 2. Outline • HDF and netCDF in ArcGIS • Time in ArcGIS • Performing Analysis • Use Cases and Applications • Script Tools •F t Future Directions Di ti
  3. 3. HDF and netCDF in ArcGIS
  4. 4. Scientific Data and ESRI • Direct support - NetCDF and HDF • THREDDS – a data server technology for multidimensional array data, integrated use by our customers • Examples using ESRI technology • National Climate Data Center • National Weather Service • National Center for Atmospheric Research • Air Force Weather • Australian Navy • Australian Bur.of Met. • UK Met Office
  5. 5. HDF4 and HDF5 Support in ArcGIS
  6. 6. Displaying MODIS LST Data HDFView ArcGIS
  7. 7. HDF4 and HDF5 Support in ArcGIS
  8. 8. What is NetCDF? NetCDF? • NetCDF (network Common Data Form) network A platform independent format for representing multi-dimensional multiarrayarray-orientated scientific data. • Self Describing: a netCDF file includes information about the data it contains. • Di Direct A t Access: a small subset of a large dataset may be accessed ll b t f l d t t b d efficiently, without first reading through all the preceding data. • Sh Sharable: one writer and multiple readers may simultaneously bl it d lti l d i lt l access the same netCDF file. NetCDF is relatively new to the GIS community but widely used by scientific communities for many years.
  9. 9. Storing Data in a netCDF File Time = 1 to 3 Y = 1 to 4 X = 1 to 4
  10. 10. NetCDF Support in ArcGIS • ArcGIS reads/writes netCDF since version 9.2 • An array based data structure for storing multidimensional data. T • N-dimensional coordinates systems • X, Y, Z, time, and other dimensions • Variables – support for multiple variables • Temperature, humidity, pressure, salinity, etc Temperature, • Geometry – implicit or explicit • Regular grid (implicit) (implicit) • Irregular grid • Points Y Z X
  11. 11. Gridded Data Regular Grid Irregular Grid
  12. 12. Ingesting netCDF data in ArcGIS • NetCDF data is accessed as Raster • Feature • Table • • Di Direct read d • Exports GIS data to netCDF
  13. 13. CF Convention Climate and Forecast (CF) Convention (CF) htt // f di ll l / Initially developed for • Climate and forecast data • Atmosphere, surface and ocean model-generated data model• Also for observational datasets • The CF conventions generalize and extend the COARDS (Cooperative cean/A Ocean/Atmosphere Research Data Service) convention. • CF i now th most widely used conventions f geospatial netCDF is the t id l d ti for ti l tCDF data. It has the best coordinate system handling.
  14. 14. NetCDF and Coordinate Systems • Geographic Coordinate Systems (GCS) X dimension units: degrees_east • Y dimension units: degrees_north • • Projected Coordinate Systems (PCS) X dimension standard_name: projection_x_coordinate standard_name: • Y dimension standard_name: projection_y_coordinate standard_name: • V i bl has a grid_mapping attribute. Variable h id i tt ib t • CF 1.4 conventions currently supports twelve predefined coordinate systems (Appendix F: Grid Mappings) Mappings) • • Undefined • If not GCS or PCS • ArcGIS writes (and recognizes) PE String as a variable attribute.
  15. 15. NetCDF Tools Toolbox: Multidimension Tools Make NetCDF Raster Layer • Make NetCDF Feature Layer • Make NetCDF Table View • Raster to NetCDF • Feature to NetCDF • Table to NetCDF • • Select by Dimension y
  16. 16. NetCDF Layer/Table Properties Raster Feature Feat re Table
  17. 17. Changing Time Slice Time = 1
  18. 18. Using NetCDF Data Behaves the same as any layer or table • Display • Same display tools for raster and feature layers will work on netCDF raster and netCDF feature layers. • Graphing • Driven by the table just like any other chart. • Animation • Multidimensional data can be animated through a dimension (e.g. time, pressure, elevation) • Analysis Tools • A netCDF layer or table will work just like any other raster layer, feature layer or table (e g create buffers around netCDF points layer, table. (e.g. points, reproject rasters, query tables, etc.) rasters,
  19. 19. Help on netCDF
  20. 20. Time in ArcGIS
  21. 21. Why visualize data through time?
  22. 22. Temporal GIS Patterns Dynamic Discrete Stationary Change something that moves something that “just happens” “j t h ” stands still but records changes change or growth • Planes • Vehicles • Animals • Satellites • Storms • Crimes • Weather Stations • Population • Lightning • Accidents • Traffic Sensors • Distribution • Fire Perimeter
  23. 23. GIS Integration of Time New Ways to Manage Visualize & Analyze Geography Manage, Multi Dimensional Data (netCDF) (netCDF) Visualize Change Vi li Ch T Real Time Sensor Network y x Files DBMS Mobile Stationary T1 Simulation Modeling
  24. 24. Time is now built-in to ArcGIS built- • Simple Temporal Mapping • Unified experience for Time • • • Configure time properties on the layer Use Time Slider to visualize temporal data is ali e Share temporal visualization TimeTime-enabled Map Services Export videos or images Generate temporal map books using ArcPy scripting • L Layer and map packages d k • • •
  25. 25. Visualizing temporal data using graphs • Create a graph using a layer or table • Create an animation in the usual way, attaching the layer or table to a time layer track • When the animation is played, p y , the graph will animate
  26. 26. Animation examples 1979
  27. 27. Performing Analysis
  28. 28. Spatial and Temporal Analysis • Several hundreds analytical tools available for raster, features, and table • Temporal Modeling p g • Looping and iteration in ModelBuilder and Python
  29. 29. Generate Rainfall Statistics • Calculates specified statistics for all time steps • Outputs a raster catalog • Optionally outputs a netCDF file
  30. 30. Generate Rainfall Statistics Table • Calculates statistics for all time steps • Outputs a table • Optionally creates a graph
  31. 31. Demo
  32. 32. Use Cases and Applications • • • • • Hydrography and METOC Branch, Royal Australian Navy Applied Science Associated, Inc. Inc. The Nature Conservancy The University of Washington The University of Southern Mississippi
  33. 33. Observations and Model Forecasts • Near real-time observations real• • • • • Weather satellite imagery Sea Surface Temperature Significant Wave Height (altimeter) Ocean Winds Moisture and Precipitation • Numerical model forecasts • Fixed domain Global, Regional, Tropical and Local atmospheric models (ACCESS) • Fixed domain Global, Regional, and Local wave models (WaveWatch 3) (WaveWatch • Global Ocean Model (BLUElink) (BLUElink) highg p • Relocatable high-resolution atmospheric and ocean models Source: Hydrography and METOC Branch, Royal Australian Navy.
  34. 34. Climatology – CCMP Winds Source: Hydrography and METOC Branch, Royal Australian Navy.
  35. 35. Climatology – CCMP Winds Source: Hydrography and METOC Branch, Royal Australian Navy.
  36. 36. Ocean Model Forecast – Custom Display Source: Hydrography and METOC Branch, Royal Australian Navy.
  37. 37. Web Services - GHRSST Source: Hydrography and METOC Branch, Royal Australian Navy.
  38. 38. Environmental Data Connector (EDC) XML DAP THREDDS Features NetCDF Java App, Unidata Rasters ArcGIS Extension Source: Applied Science Associated, Inc.
  39. 39. EDC Screen and Time Slider Toolbar Synchronize and animate time-varying data in Source: Applied Science Associated, Inc.
  40. 40. Time Series Graphs Source: Applied Science Associated, Inc.
  41. 41. Climate Wizard Developers: The Nature Conservancy, The University of Washington, and The University of Southern Mississippi
  42. 42. Climate Change Developers: The Nature Conservancy, The University of Washington, and ESRI
  43. 43. Script Tools
  44. 44. Community Developed Tools • Geoprocessing Resource Center • Marine Geospatial Ecology Tools • Developed at Duke Univ. • Over 180 tools for import management, and analysis of marine data • Australian Navy tools (not publicly available)
  45. 45. Script Tools of interest • Python is used to build custom tools for specific tasks or datasets
  46. 46. NEXRAD Geoprocessing Tools • • • • • • Currently 6 geoprocessing script tools Designed to work with NEXRAD netCDF file Can be easily modified for other datasets Customized tools for various workflows Simplify repetitive work Automate GIS processes
  47. 47. Download NEXRAD NetCDF File • Downloads netCDF file from the THREDDS server
  48. 48. Extract NEXRAD Rainfall To Points • Extracts the cell values for all time steps • Outputs a feature class
  49. 49. NEXRAD NetCDF To Raster Catalog • Creates a raster catalog • Each catalog item corresponds to a time slice in the netCDF file
  50. 50. Future Directions
  51. 51. Things to Consider… • Embrace the Common Data Model (netCDF, HDF etc.) (netCDF, • Use Data and metadata standards (OGC, CF etc) • Make your data “spatial” (by specifying geographic or a p j projected coordinate system) y ) • Clearly define workflow and requirements • Create sample tools where possible
  52. 52. Tools Under Consideration • Temporal Statistics • Get Variable Statistics Analysis • • • • Clip p Extract By Variable Extract By Dimension Append By Dimension Data Management
  53. 53. Future directions • Multidimension data management • Temporal analysis tools • Additional support for HDF5 using netCDF 4.x library • What do you need?
  54. 54. Questions? Nawajish Noman Lead Product Engineer @ i Jeff Donze ESRI Federal Business Development F d lB i D l t
  55. 55. Sharing Temporal Maps & Data with ArcGIS 10 • Publish time-aware maps time- • Export videos or images, layer and map packages • Vi li data Visualize d t Access via REST API Web API • FLEX • JavaScript • Silverlight • Time Slider web control • • 1979
  56. 56. How to store temporal data? • DATE is a special field type specific to time • GeoDatabase provides DATE • If at all possible – use DATE type • DATE field should be indexed for faster query performance • Numeric and String fields • • • • YYYY YYYYMM YYYYMMDD YYYYMMDDhhmmss YYYY/MM/DD YYYY/MM/DD hh:mm:ss • Time dimension in a netCDF variable YYYY-MM-DD YYYY-MMYYYY-MM-DD hh:mm:ss YYYY-MM-
  57. 57. Animation examples Data source: Cavalieri, D., C. Parkinson, P. Gloerson, and H.J. Zwally. 1996, updated 2005. Sea ice concentrations from Nimbus-7 SMMR and DMSP SSM/I Nimbuspassive microwave data, June to September 2001. Boulder, CO, USA: National Snow and Ice Data Center. Sea ice concentration Solar Insolation Data provided courtesy of Declan Butler - Stream flow analysis Avian Influenza
  58. 58. Atmospheric Model Forecast – Surface Wind Source: Hydrography and METOC Branch, Royal Australian Navy.