Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
 

Like this? Share it with your network

Share

Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies

on

  • 733 views

This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) ...

This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) and netCDF (network Common Data Form) are two widely used data formats for storing and manipulating scientific data. The NetCDF format also supports temporal data by using multidimensional arrays. The basic structure of data in this format and how to work with it will be covered in the context of standardized data structures and conventions. This slide will demonstrate the tools and techniques for ingesting HDF and netCDF data efficiently in ArcGIS, as well as some common workflows to employ the visualization capabilities of ArcGIS for effective animation and analysis of your data.

Statistics

Views

Total Views
733
Views on SlideShare
719
Embed Views
14

Actions

Likes
0
Downloads
20
Comments
0

5 Embeds 14

http://hdfeos.org 7
http://localhost 3
http://www.hdfeos.org 2
http://hdfdap 1
http://hdfdap.hdfgroup.uiuc.edu 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies Presentation Transcript

  • 1. Working with HDF and netCDF Data in ArcGIS Tools and Case Studies Nawajish Noman Jeff Donze HDF and HDF-EOS Workshop XIV, September 28-30, 2010, Champaign, IL HDF28-
  • 2. Outline • HDF and netCDF in ArcGIS • Time in ArcGIS • Performing Analysis • Use Cases and Applications • Script Tools •F t Future Directions Di ti
  • 3. HDF and netCDF in ArcGIS
  • 4. Scientific Data and ESRI • Direct support - NetCDF and HDF • THREDDS – a data server technology for multidimensional array data, integrated use by our customers • Examples using ESRI technology • National Climate Data Center • National Weather Service • National Center for Atmospheric Research • Air Force Weather • Australian Navy • Australian Bur.of Met. • UK Met Office
  • 5. HDF4 and HDF5 Support in ArcGIS
  • 6. Displaying MODIS LST Data HDFView ArcGIS
  • 7. HDF4 and HDF5 Support in ArcGIS http://help.arcgis.com/en/arcgisdesktop/10.0/help/
  • 8. What is NetCDF? NetCDF? • NetCDF (network Common Data Form) network A platform independent format for representing multi-dimensional multiarrayarray-orientated scientific data. • Self Describing: a netCDF file includes information about the data it contains. • Di Direct A t Access: a small subset of a large dataset may be accessed ll b t f l d t t b d efficiently, without first reading through all the preceding data. • Sh Sharable: one writer and multiple readers may simultaneously bl it d lti l d i lt l access the same netCDF file. NetCDF is relatively new to the GIS community but widely used by scientific communities for many years.
  • 9. Storing Data in a netCDF File Time = 1 to 3 Y = 1 to 4 X = 1 to 4
  • 10. NetCDF Support in ArcGIS • ArcGIS reads/writes netCDF since version 9.2 • An array based data structure for storing multidimensional data. T • N-dimensional coordinates systems • X, Y, Z, time, and other dimensions • Variables – support for multiple variables • Temperature, humidity, pressure, salinity, etc Temperature, • Geometry – implicit or explicit • Regular grid (implicit) (implicit) • Irregular grid • Points Y Z X
  • 11. Gridded Data Regular Grid Irregular Grid
  • 12. Ingesting netCDF data in ArcGIS • NetCDF data is accessed as Raster • Feature • Table • • Di Direct read d • Exports GIS data to netCDF
  • 13. CF Convention Climate and Forecast (CF) Convention (CF) http://cf-pcmdi.llnl.gov/ htt // f di ll l / Initially developed for • Climate and forecast data • Atmosphere, surface and ocean model-generated data model• Also for observational datasets • The CF conventions generalize and extend the COARDS (Cooperative cean/A Ocean/Atmosphere Research Data Service) convention. • CF i now th most widely used conventions f geospatial netCDF is the t id l d ti for ti l tCDF data. It has the best coordinate system handling.
  • 14. NetCDF and Coordinate Systems • Geographic Coordinate Systems (GCS) X dimension units: degrees_east • Y dimension units: degrees_north • • Projected Coordinate Systems (PCS) X dimension standard_name: projection_x_coordinate standard_name: • Y dimension standard_name: projection_y_coordinate standard_name: • V i bl has a grid_mapping attribute. Variable h id i tt ib t • CF 1.4 conventions currently supports twelve predefined coordinate systems (Appendix F: Grid Mappings) Mappings) • • Undefined • If not GCS or PCS • ArcGIS writes (and recognizes) PE String as a variable attribute.
  • 15. NetCDF Tools Toolbox: Multidimension Tools Make NetCDF Raster Layer • Make NetCDF Feature Layer • Make NetCDF Table View • Raster to NetCDF • Feature to NetCDF • Table to NetCDF • • Select by Dimension y
  • 16. NetCDF Layer/Table Properties Raster Feature Feat re Table
  • 17. Changing Time Slice Time = 1
  • 18. Using NetCDF Data Behaves the same as any layer or table • Display • Same display tools for raster and feature layers will work on netCDF raster and netCDF feature layers. • Graphing • Driven by the table just like any other chart. • Animation • Multidimensional data can be animated through a dimension (e.g. time, pressure, elevation) • Analysis Tools • A netCDF layer or table will work just like any other raster layer, feature layer or table (e g create buffers around netCDF points layer, table. (e.g. points, reproject rasters, query tables, etc.) rasters,
  • 19. Help on netCDF
  • 20. Time in ArcGIS
  • 21. Why visualize data through time?
  • 22. Temporal GIS Patterns Dynamic Discrete Stationary Change something that moves something that “just happens” “j t h ” stands still but records changes change or growth • Planes • Vehicles • Animals • Satellites • Storms • Crimes • Weather Stations • Population • Lightning • Accidents • Traffic Sensors • Distribution • Fire Perimeter
  • 23. GIS Integration of Time New Ways to Manage Visualize & Analyze Geography Manage, Multi Dimensional Data (netCDF) (netCDF) Visualize Change Vi li Ch T Real Time Sensor Network y x Files DBMS Mobile Stationary T1 Simulation Modeling
  • 24. Time is now built-in to ArcGIS built- • Simple Temporal Mapping • Unified experience for Time • • • Configure time properties on the layer Use Time Slider to visualize temporal data is ali e Share temporal visualization TimeTime-enabled Map Services Export videos or images Generate temporal map books using ArcPy scripting • L Layer and map packages d k • • •
  • 25. Visualizing temporal data using graphs • Create a graph using a layer or table • Create an animation in the usual way, attaching the layer or table to a time layer track • When the animation is played, p y , the graph will animate
  • 26. Animation examples 1979
  • 27. Performing Analysis
  • 28. Spatial and Temporal Analysis • Several hundreds analytical tools available for raster, features, and table • Temporal Modeling p g • Looping and iteration in ModelBuilder and Python
  • 29. Generate Rainfall Statistics • Calculates specified statistics for all time steps • Outputs a raster catalog • Optionally outputs a netCDF file
  • 30. Generate Rainfall Statistics Table • Calculates statistics for all time steps • Outputs a table • Optionally creates a graph
  • 31. Demo
  • 32. Use Cases and Applications • • • • • Hydrography and METOC Branch, Royal Australian Navy Applied Science Associated, Inc. Inc. The Nature Conservancy The University of Washington The University of Southern Mississippi
  • 33. Observations and Model Forecasts • Near real-time observations real• • • • • Weather satellite imagery Sea Surface Temperature Significant Wave Height (altimeter) Ocean Winds Moisture and Precipitation • Numerical model forecasts • Fixed domain Global, Regional, Tropical and Local atmospheric models (ACCESS) • Fixed domain Global, Regional, and Local wave models (WaveWatch 3) (WaveWatch • Global Ocean Model (BLUElink) (BLUElink) highg p • Relocatable high-resolution atmospheric and ocean models Source: Hydrography and METOC Branch, Royal Australian Navy.
  • 34. Climatology – CCMP Winds Source: Hydrography and METOC Branch, Royal Australian Navy.
  • 35. Climatology – CCMP Winds Source: Hydrography and METOC Branch, Royal Australian Navy.
  • 36. Ocean Model Forecast – Custom Display Source: Hydrography and METOC Branch, Royal Australian Navy.
  • 37. Web Services - GHRSST Source: Hydrography and METOC Branch, Royal Australian Navy.
  • 38. Environmental Data Connector (EDC) XML DAP THREDDS Features NetCDF Java App, Unidata Rasters ArcGIS Extension Source: Applied Science Associated, Inc.
  • 39. EDC Screen and Time Slider Toolbar Synchronize and animate time-varying data in http://www.pfeg.noaa.gov/products/EDC/ http://www.asascience.com/TimeSlider/index.htm Source: Applied Science Associated, Inc.
  • 40. Time Series Graphs Source: Applied Science Associated, Inc.
  • 41. Climate Wizard http://www.climatewizard.org/ Developers: The Nature Conservancy, The University of Washington, and The University of Southern Mississippi
  • 42. Climate Change http://tncclimate.esri.com/TNC/ClimateChange.html Developers: The Nature Conservancy, The University of Washington, and ESRI
  • 43. Script Tools
  • 44. Community Developed Tools • Geoprocessing Resource Center http://resources.arcgis.com/geoprocessing/ • Marine Geospatial Ecology Tools • Developed at Duke Univ. • Over 180 tools for import management, and analysis of marine data • Australian Navy tools (not publicly available)
  • 45. Script Tools of interest • Python is used to build custom tools for specific tasks or datasets
  • 46. NEXRAD Geoprocessing Tools • • • • • • Currently 6 geoprocessing script tools Designed to work with NEXRAD netCDF file Can be easily modified for other datasets Customized tools for various workflows Simplify repetitive work Automate GIS processes
  • 47. Download NEXRAD NetCDF File • Downloads netCDF file from the THREDDS server
  • 48. Extract NEXRAD Rainfall To Points • Extracts the cell values for all time steps • Outputs a feature class
  • 49. NEXRAD NetCDF To Raster Catalog • Creates a raster catalog • Each catalog item corresponds to a time slice in the netCDF file
  • 50. Future Directions
  • 51. Things to Consider… • Embrace the Common Data Model (netCDF, HDF etc.) (netCDF, • Use Data and metadata standards (OGC, CF etc) • Make your data “spatial” (by specifying geographic or a p j projected coordinate system) y ) • Clearly define workflow and requirements • Create sample tools where possible
  • 52. Tools Under Consideration • Temporal Statistics • Get Variable Statistics Analysis • • • • Clip p Extract By Variable Extract By Dimension Append By Dimension Data Management
  • 53. Future directions • Multidimension data management • Temporal analysis tools • Additional support for HDF5 using netCDF 4.x library • What do you need?
  • 54. Questions? Nawajish Noman Lead Product Engineer nnoman@esri.com @ i Jeff Donze ESRI Federal Business Development F d lB i D l t jdonze@esri.com
  • 55. Sharing Temporal Maps & Data with ArcGIS 10 • Publish time-aware maps time- • Export videos or images, layer and map packages • Vi li data Visualize d t Access via REST API Web API • FLEX • JavaScript • Silverlight • Time Slider web control • • 1979
  • 56. How to store temporal data? • DATE is a special field type specific to time • GeoDatabase provides DATE • If at all possible – use DATE type • DATE field should be indexed for faster query performance • Numeric and String fields • • • • YYYY YYYYMM YYYYMMDD YYYYMMDDhhmmss YYYY/MM/DD YYYY/MM/DD hh:mm:ss • Time dimension in a netCDF variable YYYY-MM-DD YYYY-MMYYYY-MM-DD hh:mm:ss YYYY-MM-
  • 57. Animation examples Data source: Cavalieri, D., C. Parkinson, P. Gloerson, and H.J. Zwally. 1996, updated 2005. Sea ice concentrations from Nimbus-7 SMMR and DMSP SSM/I Nimbuspassive microwave data, June to September 2001. Boulder, CO, USA: National Snow and Ice Data Center. Sea ice concentration Solar Insolation Data provided courtesy of Declan Butler - http://declanbutler.info/blog/ Stream flow analysis Avian Influenza
  • 58. Atmospheric Model Forecast – Surface Wind Source: Hydrography and METOC Branch, Royal Australian Navy.