Downloaded 232 times
Uploaded byThe HDF-EOS Tools and Information Center
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
The document discusses the integration of HDF and NetCDF data formats into ArcGIS, focusing on tools, applications, and methods for analyzing scientific data. It outlines features such as multidimensional data support, temporal visualization, and case studies from various organizations and agencies. Future directions include enhancing data management and analysis tools while promoting standards and workflows in utilizing these data formats.
More Related Content
Similar to Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
More from The HDF-EOS Tools and Information Center
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
- 1. Working with HDFand netCDF Data in ArcGIS Tools and Case Studies Nawajish Noman Jeff Donze HDF and HDF-EOS Workshop XIV, September 28-30, 2010, Champaign, IL HDF28-
- 2. Outline • HDF andnetCDF in ArcGIS • Time in ArcGIS • Performing Analysis • Use Cases and Applications • Script Tools •F t Future Directions Di ti
- 3. HDF and netCDFin ArcGIS
- 4. Scientific Data andESRI • Direct support - NetCDF and HDF • THREDDS – a data server technology for multidimensional array data, integrated use by our customers • Examples using ESRI technology • National Climate Data Center • National Weather Service • National Center for Atmospheric Research • Air Force Weather • Australian Navy • Australian Bur.of Met. • UK Met Office
- 5. HDF4 and HDF5Support in ArcGIS
- 6. Displaying MODIS LSTData HDFView ArcGIS
- 7. HDF4 and HDF5Support in ArcGIS http://help.arcgis.com/en/arcgisdesktop/10.0/help/
- 8. What is NetCDF? NetCDF? •NetCDF (network Common Data Form) network A platform independent format for representing multi-dimensional multiarrayarray-orientated scientific data. • Self Describing: a netCDF file includes information about the data it contains. • Di Direct A t Access: a small subset of a large dataset may be accessed ll b t f l d t t b d efficiently, without first reading through all the preceding data. • Sh Sharable: one writer and multiple readers may simultaneously bl it d lti l d i lt l access the same netCDF file. NetCDF is relatively new to the GIS community but widely used by scientific communities for many years.
- 9. Storing Data ina netCDF File Time = 1 to 3 Y = 1 to 4 X = 1 to 4
- 10. NetCDF Support inArcGIS • ArcGIS reads/writes netCDF since version 9.2 • An array based data structure for storing multidimensional data. T • N-dimensional coordinates systems • X, Y, Z, time, and other dimensions • Variables – support for multiple variables • Temperature, humidity, pressure, salinity, etc Temperature, • Geometry – implicit or explicit • Regular grid (implicit) (implicit) • Irregular grid • Points Y Z X
- 11.
- 12. Ingesting netCDF datain ArcGIS • NetCDF data is accessed as Raster • Feature • Table • • Di Direct read d • Exports GIS data to netCDF
- 13. CF Convention Climate andForecast (CF) Convention (CF) http://cf-pcmdi.llnl.gov/ htt // f di ll l / Initially developed for • Climate and forecast data • Atmosphere, surface and ocean model-generated data model• Also for observational datasets • The CF conventions generalize and extend the COARDS (Cooperative cean/A Ocean/Atmosphere Research Data Service) convention. • CF i now th most widely used conventions f geospatial netCDF is the t id l d ti for ti l tCDF data. It has the best coordinate system handling.
- 14. NetCDF and CoordinateSystems • Geographic Coordinate Systems (GCS) X dimension units: degrees_east • Y dimension units: degrees_north • • Projected Coordinate Systems (PCS) X dimension standard_name: projection_x_coordinate standard_name: • Y dimension standard_name: projection_y_coordinate standard_name: • V i bl has a grid_mapping attribute. Variable h id i tt ib t • CF 1.4 conventions currently supports twelve predefined coordinate systems (Appendix F: Grid Mappings) Mappings) • • Undefined • If not GCS or PCS • ArcGIS writes (and recognizes) PE String as a variable attribute.
- 15. NetCDF Tools Toolbox: MultidimensionTools Make NetCDF Raster Layer • Make NetCDF Feature Layer • Make NetCDF Table View • Raster to NetCDF • Feature to NetCDF • Table to NetCDF • • Select by Dimension y
- 16.
- 17.
- 18. Using NetCDF Data Behavesthe same as any layer or table • Display • Same display tools for raster and feature layers will work on netCDF raster and netCDF feature layers. • Graphing • Driven by the table just like any other chart. • Animation • Multidimensional data can be animated through a dimension (e.g. time, pressure, elevation) • Analysis Tools • A netCDF layer or table will work just like any other raster layer, feature layer or table (e g create buffers around netCDF points layer, table. (e.g. points, reproject rasters, query tables, etc.) rasters,
- 19.
- 20.
- 21. Why visualize datathrough time?
- 22. Temporal GIS Patterns Dynamic Discrete Stationary Change somethingthat moves something that “just happens” “j t h ” stands still but records changes change or growth • Planes • Vehicles • Animals • Satellites • Storms • Crimes • Weather Stations • Population • Lightning • Accidents • Traffic Sensors • Distribution • Fire Perimeter
- 23. GIS Integration ofTime New Ways to Manage Visualize & Analyze Geography Manage, Multi Dimensional Data (netCDF) (netCDF) Visualize Change Vi li Ch T Real Time Sensor Network y x Files DBMS Mobile Stationary T1 Simulation Modeling
- 24. Time is nowbuilt-in to ArcGIS built- • Simple Temporal Mapping • Unified experience for Time • • • Configure time properties on the layer Use Time Slider to visualize temporal data is ali e Share temporal visualization TimeTime-enabled Map Services Export videos or images Generate temporal map books using ArcPy scripting • L Layer and map packages d k • • •
- 25. Visualizing temporal datausing graphs • Create a graph using a layer or table • Create an animation in the usual way, attaching the layer or table to a time layer track • When the animation is played, p y , the graph will animate
- 26.
- 27.
- 28. Spatial and TemporalAnalysis • Several hundreds analytical tools available for raster, features, and table • Temporal Modeling p g • Looping and iteration in ModelBuilder and Python
- 29. Generate Rainfall Statistics •Calculates specified statistics for all time steps • Outputs a raster catalog • Optionally outputs a netCDF file
- 30. Generate Rainfall StatisticsTable • Calculates statistics for all time steps • Outputs a table • Optionally creates a graph
- 31.
- 32. Use Cases andApplications • • • • • Hydrography and METOC Branch, Royal Australian Navy Applied Science Associated, Inc. Inc. The Nature Conservancy The University of Washington The University of Southern Mississippi
- 33. Observations and ModelForecasts • Near real-time observations real• • • • • Weather satellite imagery Sea Surface Temperature Significant Wave Height (altimeter) Ocean Winds Moisture and Precipitation • Numerical model forecasts • Fixed domain Global, Regional, Tropical and Local atmospheric models (ACCESS) • Fixed domain Global, Regional, and Local wave models (WaveWatch 3) (WaveWatch • Global Ocean Model (BLUElink) (BLUElink) highg p • Relocatable high-resolution atmospheric and ocean models Source: Hydrography and METOC Branch, Royal Australian Navy.
- 34. Climatology – CCMPWinds Source: Hydrography and METOC Branch, Royal Australian Navy.
- 35. Climatology – CCMPWinds Source: Hydrography and METOC Branch, Royal Australian Navy.
- 36. Ocean Model Forecast– Custom Display Source: Hydrography and METOC Branch, Royal Australian Navy.
- 37. Web Services -GHRSST Source: Hydrography and METOC Branch, Royal Australian Navy.
- 38. Environmental Data Connector(EDC) XML DAP THREDDS Features NetCDF Java App, Unidata Rasters ArcGIS Extension Source: Applied Science Associated, Inc.
- 39. EDC Screen andTime Slider Toolbar Synchronize and animate time-varying data in http://www.pfeg.noaa.gov/products/EDC/ http://www.asascience.com/TimeSlider/index.htm Source: Applied Science Associated, Inc.
- 40. Time Series Graphs Source:Applied Science Associated, Inc.
- 41. Climate Wizard http://www.climatewizard.org/ Developers: TheNature Conservancy, The University of Washington, and The University of Southern Mississippi
- 42. Climate Change http://tncclimate.esri.com/TNC/ClimateChange.html Developers: TheNature Conservancy, The University of Washington, and ESRI
- 43.
- 44. Community Developed Tools •Geoprocessing Resource Center http://resources.arcgis.com/geoprocessing/ • Marine Geospatial Ecology Tools • Developed at Duke Univ. • Over 180 tools for import management, and analysis of marine data • Australian Navy tools (not publicly available)
- 45. Script Tools ofinterest • Python is used to build custom tools for specific tasks or datasets
- 46. NEXRAD Geoprocessing Tools • • • • • • Currently6 geoprocessing script tools Designed to work with NEXRAD netCDF file Can be easily modified for other datasets Customized tools for various workflows Simplify repetitive work Automate GIS processes
- 47. Download NEXRAD NetCDFFile • Downloads netCDF file from the THREDDS server
- 48. Extract NEXRAD RainfallTo Points • Extracts the cell values for all time steps • Outputs a feature class
- 49. NEXRAD NetCDF ToRaster Catalog • Creates a raster catalog • Each catalog item corresponds to a time slice in the netCDF file
- 50.
- 51. Things to Consider… •Embrace the Common Data Model (netCDF, HDF etc.) (netCDF, • Use Data and metadata standards (OGC, CF etc) • Make your data “spatial” (by specifying geographic or a p j projected coordinate system) y ) • Clearly define workflow and requirements • Create sample tools where possible
- 52. Tools Under Consideration •Temporal Statistics • Get Variable Statistics Analysis • • • • Clip p Extract By Variable Extract By Dimension Append By Dimension Data Management
- 53. Future directions • Multidimensiondata management • Temporal analysis tools • Additional support for HDF5 using netCDF 4.x library • What do you need?
- 54. Questions? Nawajish Noman Lead ProductEngineer nnoman@esri.com @ i Jeff Donze ESRI Federal Business Development F d lB i D l t jdonze@esri.com
- 55. Sharing Temporal Maps& Data with ArcGIS 10 • Publish time-aware maps time- • Export videos or images, layer and map packages • Vi li data Visualize d t Access via REST API Web API • FLEX • JavaScript • Silverlight • Time Slider web control • • 1979
- 56. How to storetemporal data? • DATE is a special field type specific to time • GeoDatabase provides DATE • If at all possible – use DATE type • DATE field should be indexed for faster query performance • Numeric and String fields • • • • YYYY YYYYMM YYYYMMDD YYYYMMDDhhmmss YYYY/MM/DD YYYY/MM/DD hh:mm:ss • Time dimension in a netCDF variable YYYY-MM-DD YYYY-MMYYYY-MM-DD hh:mm:ss YYYY-MM-
- 57. Animation examples Data source:Cavalieri, D., C. Parkinson, P. Gloerson, and H.J. Zwally. 1996, updated 2005. Sea ice concentrations from Nimbus-7 SMMR and DMSP SSM/I Nimbuspassive microwave data, June to September 2001. Boulder, CO, USA: National Snow and Ice Data Center. Sea ice concentration Solar Insolation Data provided courtesy of Declan Butler - http://declanbutler.info/blog/ Stream flow analysis Avian Influenza
- 58. Atmospheric Model Forecast– Surface Wind Source: Hydrography and METOC Branch, Royal Australian Navy.