HDF and HDF-EOS Workshop XX (2017)

1. JL1-SESIP-0717
Guided Tour of
Pythonian Museum
H. Joe Lee (hyoklee@hdfgroup.org)
The HDF Group
This work was supported by NASA/GSFC under
Raytheon Co. contract number NNG15HZ39C

2. JL1-SESIP-0717
2
Welcome to Pythonian Museum!
https://commons.wikimedia.org/wiki/File:Smithsonian_Building_NR.jpg

3. JL1-SESIP-0717
3
Your tour guide
• is the maintainer of hdfeos.org website.
• loves diverse NASA HDF products.
• has created 170+ Python examples so far.

4. JL1-SESIP-0717
4
Where is the museum?
http://hdfeos.org/zoo
Let’s
Go!
Public use graphics
https://www.autodraw.com/artists

5. JL1-SESIP-0717
5
The Main Entrance
• List of NASA Data Centers
• List of NASA HDF Products
Click on any product!

6. JL1-SESIP-0717
6
Pythonian tour has 4 packages
• pyhdf – for HDF4
• h5py – for HDF5
• netCDF4 – for HDF4/HDF5
• gdal – for HDF-EOS2/HDF-EOS5

7. JL1-SESIP-0717
7
Why 4 packages?

8. JL1-SESIP-0717
8
How about other packages?

9. JL1-SESIP-0717
9
What to focus on during tour
• Does product have lat/lon dataset?
• If not, are they stored in different product
or metadata or document?
• Bit packing / scale & offset handling
• Visualization tips on world map with
different projections
• Memory / performance issue

10. JL1-SESIP-0717
10
GES DISC AIRS Grid
• HDF-EOS2 – 1D Lat/Lon variables
• pyhdf / netCDF4
• Equidistant Cylindrical plot
# Read geolocation dataset.
lat = hdf.select(‘Latitude’)

11. JL1-SESIP-0717
11
GES DISC AIRS Swath
• HDF-EOS2 – 2D Lat/Lon variables
• pyhdf / netCDF4
• Polar Stereographic plot
# Read geolocation dataset.
lat = hdf.select('Latitude')
latitude = lat[:,:]
# Polar stereo
m =
Basemap(projection='npstere',
resolution='l'
boundinglat=65, lon_0 = 180)

12. JL1-SESIP-0717
12
GES DISC TRMM Swath v7
• HDF4 – pyhdf / netCDF4
• Lat/Lon in one 3D dataset - subsetting
• Longitude shifting for pcolormesh()
# There is a wrap-around effect to deal with.
longitude[longitude < -165] += 360

13. JL1-SESIP-0717
13
GES DISC TRMM Grid
• Lat/Lon are calculated from documentation.
# http://disc.sci.gsfc.nasa.gov/additional/faq/precipitation_faq.shtml
latitude = np.arange(-49.875, 49.875, 0.249375)
longitude = np.arange(-179.875, 179.876, 0.25)

14. JL1-SESIP-0717
14
GES DISC OMI Grid
• HDF-EOS5 - h5py (Python3)
• Lat/lon are calculated from StructMetadata
# There is no geolocation data, so construct it ourselves.
longitude = np.arange(0., 1440.0) * 0.25 - 180 + 0.125
latitude = np.arange(0., 720.0) * 0.25 - 90 + 0.125

15. JL1-SESIP-0717
15
GES DISC MLS Swath
• HDF-EOS5 / h5py
• netCDF-4 will not work due to object reference
• Vertical profile line plot –plot()
• Toolkit Internal Time (TAI) handling
# Handle TAI.
timebase =
datetime.datetime(1993, 1,
1, 0, 0, 0) +
datetime.timedelta(seconds
=time[399])

16. JL1-SESIP-0717
16
GES DISC HIRDLS ZA
• HDF-EOS5 - h5py / netCDF4
• Zonal Average plot - contourf()
• Log scale in Y-axis
# Apply log scale along
the y-axis to get a better
image.
lev = np.log10(lev)

17. JL1-SESIP-0717
17
GES DISC GOSAT/ACOS
• HDF5 – h5py
• 1D lat/lon swath
• Trajectory - scatter()
• Multiple plots
• Orthographic projection map

18. JL1-SESIP-0717
18
LAADS MOD06 Swath
• HDF-EOS2 – pyhdf / netCDF4
• Lat/lon from MOD03 product
• Scale / offset / valid range
• S. Pole Stereographic projection
# Read geolocation dataset
from MOD03 product.
hdf = SD(FILE_NAME, SDC.READ)
hdf_geo = SD(GEO_FILE_NAME,
SDC.READ)
lat =
hdf_geo.select('Latitude')

19. JL1-SESIP-0717
19
LP DAAC MOD09GA Grid
• HDF-EOS2 – pyhdf / gdal
• Lat/lon in Sinusoidal Projection
• mpl_toolkits.basemap.pyproj
# In basemap, the sinusoidal
projection is global, so we
won't use it.
# Instead we'll convert the
grid back to lat/lons.
sinu = pyproj.Proj("+proj=sinu
+R=6371007.181 +nadgrids=@null
+wktext")
wgs84 =
pyproj.Proj("+init=EPSG:4326")
lon, lat=
pyproj.transform(sinu, wgs84,
xv, yv)

20. JL1-SESIP-0717
20
MEaSURES VIP GRID
• HDF-EOS2 – pyhdf / netCDF4 / gdal
• StructMetadata parsing
• Huge dataset – sub sampling for visualization
# Scale down the data by a factor of 6 so that low-memory machines can
handle it.
data = data[::6, ::6]

21. JL1-SESIP-0717
21
ASDC CALIPSO
• HDF4 – pyhdf / netCDF4
• Bitmask / Complex subsetting
• Discrete colorbar
# You can visualize other
blocks by changing subset
parameters.
data2d = data[3500:3999,
0:164] # 20.2km to
30.1km
data2d = data[3500:3999,
165:1164] # 8.2km to
20.2km
data2d = data[3500:4000,
1165:] # -0.5km to 8.2km

22. JL1-SESIP-0717
22
More Examples?
• NASA Developers Portal – PyDAP + PyCMR
https://developer.earthdata.nasa.gov/opendap/cms-ch4-flx-na

23. JL1-SESIP-0717
23
This work was supported by
NASA/GSFC under Raytheon Co.
contract number NNG15HZ39C