Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Are from Mars, Tools Are from Venus

218 views

Published on

HDF and HDF-EOS Workshop XX (2017)

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Data Are from Mars, Tools Are from Venus

  1. 1. JL2-SESIP-0717 Data Are from Mars, Tools Are from Venus H. Joe Lee (hyoklee@hdfgroup.org) The HDF Group This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C All images used in this presentation are from autodraw.com for public use.
  2. 2. JL2-SESIP-0717 2 No “Earth” in title? “Men Are from Mars, Women are from Venus” - John Gray
  3. 3. JL2-SESIP-0717 3 Are data from Mars? Why can’t I use Earth tools? Correct geo-referencing
  4. 4. JL2-SESIP-0717 4 Are tools from Venus? Why can’t I open Earth data?
  5. 5. JL2-SESIP-0717 5 Data Producers from Mars • I want my data slim and efficient. • How can I save money in managing data? Tool Developers from Venus • I make my tool work for popular data first. • Can I make money by supporting your data?
  6. 6. JL2-SESIP-0717 6 Result: Frustrated Users
  7. 7. JL2-SESIP-0717 7 We (HDFEOS.org) can help.
  8. 8. JL2-SESIP-0717 8 • Hierarchical Data Format (HDF) • Network Common Data Format (netCDF) • Geospatial Tagged Image File Format (GeoTIFF) • Keyhole Markup Language (KML) / zipped KML (KMZ) • Comma-separated values (CSV) • etc. We identify gaps in File Formats
  9. 9. JL2-SESIP-0717 9 • hdf • netcdf-C • netcdf-Java • HDF – Earth Observing System (hdf-eos) • Climate and Forecast Metadata (CF) conventions • Geospatial Data Abstraction Library (GDAL) • etc. We identify gaps in Libraries
  10. 10. JL2-SESIP-0717 10 • Microsoft Excel • Esri ArcGIS • Google Earth • MATLAB • Python • Interactive Data Language (IDL) • Panoply • Integrated Data Viewer (IDV) • HDFView • h5dump • Etc. We identify gaps in Tools
  11. 11. JL2-SESIP-0717 11 • Open-source Project for a Network Data Access Protocol (OPeNDAP) • Web Map Service (WMS) • Web Map Tile Service (WMTS) • Web Coverage Service (WCS) • etc. We identify gaps in Services
  12. 12. JL2-SESIP-0717 12 • File conversion • Libraries and tools usage • NASA HDF product specific examples • Demo services (e.g., Hyrax*, THREDDS**) AND we provide Solutions… *Hyrax is the data server from OPeNDAP. **Thematic Real-time Environmental Distributed Data Services
  13. 13. JL2-SESIP-0717 13 • Make HDF5 data work with – GDAL – netCDF – Hyrax/THREDDS • Don’t forget a few key CF conventions. • Follow DIWG* recommendations. Suggestions for data producers *Data Interoperability Working Group
  14. 14. JL2-SESIP-0717 14 • Download and test NASA HDF products. • Support them natively. • Support augmentation. – VRT* in GDAL – NcML** in netCDF • Support 3D visualization for data in the air. Suggestions for tool developers *Virtual Dataset in XML format **netCDF Markup Language
  15. 15. JL2-SESIP-0717 15 3D Visualization
  16. 16. JL2-SESIP-0717 16 • Try OPeNDAP first. CSV may be enough. • Try netCDF conversion / augmentation. • Correct metadata with NcML / VRT. • Try GEE* instead of GDAL. • Use CMR** wisely. Suggestions for end-users *GDAL Enhancement for ESDIS project ** Core Metadata Repository
  17. 17. JL2-SESIP-0717 17 • Hadoop / Spark (streaming) / Dask • Parquet / Arrow • Elastic Search / Kibana How about Big (fast) data?
  18. 18. JL2-SESIP-0717 18 Streaming Swath Hadoop + Spark Streaming + Elasticsearch & Kibana S3 + AWS Lambda + Elasticsearch & Kibana
  19. 19. JL2-SESIP-0717 19 • scikit-learn / keras / h2o.ai • Please contact us at eoshelp@hdfgroup.org if you’d like to see examples on machine learning. Future: (Deep) Machine Learning?
  20. 20. JL2-SESIP-0717 20 Deep Learning: Amazon Rekognition
  21. 21. JL2-SESIP-0717 21 Alexa, what does MODIS see now? Alexa: I can recognize “Alps.”
  22. 22. JL2-SESIP-0717 22 This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C

×