Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Guided Tour of Pythonian Museum

202 views

Published on

HDF and HDF-EOS Workshop XX (2017)

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Guided Tour of Pythonian Museum

  1. 1. JL1-SESIP-0717 Guided Tour of Pythonian Museum H. Joe Lee (hyoklee@hdfgroup.org) The HDF Group This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C
  2. 2. JL1-SESIP-0717 2 Welcome to Pythonian Museum! https://commons.wikimedia.org/wiki/File:Smithsonian_Building_NR.jpg
  3. 3. JL1-SESIP-0717 3 Your tour guide • is the maintainer of hdfeos.org website. • loves diverse NASA HDF products. • has created 170+ Python examples so far.
  4. 4. JL1-SESIP-0717 4 Where is the museum? http://hdfeos.org/zoo Let’s Go! Public use graphics https://www.autodraw.com/artists
  5. 5. JL1-SESIP-0717 5 The Main Entrance • List of NASA Data Centers • List of NASA HDF Products Click on any product!
  6. 6. JL1-SESIP-0717 6 Pythonian tour has 4 packages • pyhdf – for HDF4 • h5py – for HDF5 • netCDF4 – for HDF4/HDF5 • gdal – for HDF-EOS2/HDF-EOS5
  7. 7. JL1-SESIP-0717 7 Why 4 packages?
  8. 8. JL1-SESIP-0717 8 How about other packages?
  9. 9. JL1-SESIP-0717 9 What to focus on during tour • Does product have lat/lon dataset? • If not, are they stored in different product or metadata or document? • Bit packing / scale & offset handling • Visualization tips on world map with different projections • Memory / performance issue
  10. 10. JL1-SESIP-0717 10 GES DISC AIRS Grid • HDF-EOS2 – 1D Lat/Lon variables • pyhdf / netCDF4 • Equidistant Cylindrical plot # Read geolocation dataset. lat = hdf.select(‘Latitude’)
  11. 11. JL1-SESIP-0717 11 GES DISC AIRS Swath • HDF-EOS2 – 2D Lat/Lon variables • pyhdf / netCDF4 • Polar Stereographic plot # Read geolocation dataset. lat = hdf.select('Latitude') latitude = lat[:,:] # Polar stereo m = Basemap(projection='npstere', resolution='l' boundinglat=65, lon_0 = 180)
  12. 12. JL1-SESIP-0717 12 GES DISC TRMM Swath v7 • HDF4 – pyhdf / netCDF4 • Lat/Lon in one 3D dataset - subsetting • Longitude shifting for pcolormesh() # There is a wrap-around effect to deal with. longitude[longitude < -165] += 360
  13. 13. JL1-SESIP-0717 13 GES DISC TRMM Grid • Lat/Lon are calculated from documentation. # http://disc.sci.gsfc.nasa.gov/additional/faq/precipitation_faq.shtml latitude = np.arange(-49.875, 49.875, 0.249375) longitude = np.arange(-179.875, 179.876, 0.25)
  14. 14. JL1-SESIP-0717 14 GES DISC OMI Grid • HDF-EOS5 - h5py (Python3) • Lat/lon are calculated from StructMetadata # There is no geolocation data, so construct it ourselves. longitude = np.arange(0., 1440.0) * 0.25 - 180 + 0.125 latitude = np.arange(0., 720.0) * 0.25 - 90 + 0.125
  15. 15. JL1-SESIP-0717 15 GES DISC MLS Swath • HDF-EOS5 / h5py • netCDF-4 will not work due to object reference • Vertical profile line plot –plot() • Toolkit Internal Time (TAI) handling # Handle TAI. timebase = datetime.datetime(1993, 1, 1, 0, 0, 0) + datetime.timedelta(seconds =time[399])
  16. 16. JL1-SESIP-0717 16 GES DISC HIRDLS ZA • HDF-EOS5 - h5py / netCDF4 • Zonal Average plot - contourf() • Log scale in Y-axis # Apply log scale along the y-axis to get a better image. lev = np.log10(lev)
  17. 17. JL1-SESIP-0717 17 GES DISC GOSAT/ACOS • HDF5 – h5py • 1D lat/lon swath • Trajectory - scatter() • Multiple plots • Orthographic projection map
  18. 18. JL1-SESIP-0717 18 LAADS MOD06 Swath • HDF-EOS2 – pyhdf / netCDF4 • Lat/lon from MOD03 product • Scale / offset / valid range • S. Pole Stereographic projection # Read geolocation dataset from MOD03 product. hdf = SD(FILE_NAME, SDC.READ) hdf_geo = SD(GEO_FILE_NAME, SDC.READ) lat = hdf_geo.select('Latitude')
  19. 19. JL1-SESIP-0717 19 LP DAAC MOD09GA Grid • HDF-EOS2 – pyhdf / gdal • Lat/lon in Sinusoidal Projection • mpl_toolkits.basemap.pyproj # In basemap, the sinusoidal projection is global, so we won't use it. # Instead we'll convert the grid back to lat/lons. sinu = pyproj.Proj("+proj=sinu +R=6371007.181 +nadgrids=@null +wktext") wgs84 = pyproj.Proj("+init=EPSG:4326") lon, lat= pyproj.transform(sinu, wgs84, xv, yv)
  20. 20. JL1-SESIP-0717 20 MEaSURES VIP GRID • HDF-EOS2 – pyhdf / netCDF4 / gdal • StructMetadata parsing • Huge dataset – sub sampling for visualization # Scale down the data by a factor of 6 so that low-memory machines can handle it. data = data[::6, ::6]
  21. 21. JL1-SESIP-0717 21 ASDC CALIPSO • HDF4 – pyhdf / netCDF4 • Bitmask / Complex subsetting • Discrete colorbar # You can visualize other blocks by changing subset parameters. data2d = data[3500:3999, 0:164] # 20.2km to 30.1km data2d = data[3500:3999, 165:1164] # 8.2km to 20.2km data2d = data[3500:4000, 1165:] # -0.5km to 8.2km
  22. 22. JL1-SESIP-0717 22 More Examples? • NASA Developers Portal – PyDAP + PyCMR https://developer.earthdata.nasa.gov/opendap/cms-ch4-flx-na
  23. 23. JL1-SESIP-0717 23 This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C

×