Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fr1T101-Kuo-20110729 IGARSS ESC.pptx

493 views

Published on

  • Be the first to comment

  • Be the first to like this

Fr1T101-Kuo-20110729 IGARSS ESC.pptx

  1. 1. A PROPOSED EARTH SCIENCE COLLABORATORY <br />K-S Kuo1,2, Chris Lynnes1, Rahul Ramachandran3<br />1NASAGoddard Space Flight Center, USA<br />2Caelum Research Corporation, USA<br />3University of Alabama-Huntsville, USA<br />7/27/11<br />1<br />IGARSS 2011, Vancouver, Canada<br />
  2. 2. Why ESC?<br />7/27/11<br />2<br />IGARSS 2011, Vancouver, Canada<br />Data Intensive Science<br />Many forms and sources of data<br />In situ measurements<br />Remote sensing observations<br />Model simulations<br />Large volumes of data<br />Effectiveness as a scientist<br />Increasing proportion of effort in data management<br />Threatening:<br />Reproducibility<br />Correctness<br />Productivity<br />
  3. 3. What is an ESC?<br />Vision of arich model development/simulation and data analysis environment that:<br />Provides access to various Earth Science models<br />Facilitates model and analysis software development<br />Provides access across a wide spectrum of Earth Science data<br />Provides a diverse set of science analysis services and tools<br />Supports the application of services and tools to data<br />Supports collaboration, i.e. sharing of data, tools and results<br />Supports discovery and publication of all science artifacts<br />7/27/11<br />3<br />IGARSS 2011, Vancouver, Canada<br />Basically, a new and natural place for Earth scientists to conduct their work and collaborate with others!<br />
  4. 4. 7/27/11<br />4<br />IGARSS 2011, Vancouver, Canada<br />The Situation TodayIslands of data and services with selective connectivity<br />Data Center A<br />Data Center C<br />Data Center B<br />
  5. 5. High-Level View<br />7/27/11<br />5<br />IGARSS 2011, Vancouver, Canada<br />Cyberinfrastructure<br />Laboratory Notebook<br />Workflow<br />Mediator<br />Tool Library<br />Data Library<br />Data Centers<br />
  6. 6. 7/27/11<br />6<br />IGARSS 2011, Vancouver, Canada<br />Tool Library<br /><ul><li>Discovery
  7. 7. Social
  8. 8. Sharing
  9. 9. Tagging
  10. 10. Discussion
  11. 11. Configuration Management
  12. 12. Testing
  13. 13. Versioning</li></ul>Packager<br /><ul><li>autoconf
  14. 14. RPM
  15. 15. Web wrapper</li></ul>Provisioned<br /><ul><li>GrADS
  16. 16. IDL
  17. 17. MatLab
  18. 18. ncl
  19. 19. nco
  20. 20. cdat</li></ul>Contributed<br /><ul><li>[Tool 1]
  21. 21. [Tool 2]
  22. 22. [Tool 3]
  23. 23. [Tool 4]
  24. 24. [Tool 5]
  25. 25. …</li></ul>Community<br /><ul><li>Quality filter
  26. 26. Coincidence
  27. 27. Feature detection
  28. 28. Event service
  29. 29. Visualization</li></ul>Personal<br /><ul><li>[Tool 1]
  30. 30. [Tool 2]
  31. 31. [Tool 3]
  32. 32. [Tool 4]
  33. 33. [Tool 5]
  34. 34. …</li></li></ul><li>7/27/11<br />7<br />IGARSS 2011, Vancouver, Canada<br />Data Library<br /><ul><li>Cache
  35. 35. Discovery
  36. 36. Social
  37. 37. Sharing
  38. 38. Tagging
  39. 39. Discussion
  40. 40. Configuration Management
  41. 41. Testing
  42. 42. Versioning</li></ul>Packager<br /><ul><li>data probe
  43. 43. format check
  44. 44. metadata wizard</li></ul>Provisioned<br /><ul><li>EOSDIS</li></ul>Contributed<br /><ul><li>[Dataset 1]
  45. 45. [Dataset 2]
  46. 46. [Dataset 3]
  47. 47. [Dataset 4]
  48. 48. [Dataset 5]
  49. 49. …</li></ul>Community<br /><ul><li>Field campaigns
  50. 50. MEaSUREs
  51. 51. ACCESS
  52. 52. Validation</li></ul>Personal<br /><ul><li>[Dataset 1]
  53. 53. [Dataset 2]
  54. 54. [Dataset 3]
  55. 55. …</li></li></ul><li>7/27/11<br />8<br />IGARSS 2011, Vancouver, Canada<br />Workflow Library<br /><ul><li>Discovery
  56. 56. Social
  57. 57. Sharing
  58. 58. Tagging
  59. 59. Discussion
  60. 60. Configuration Management
  61. 61. Testing
  62. 62. Versioning</li></ul>Packager<br /><ul><li>Workflow editor</li></ul>Provisioned<br /><ul><li>Processing Algorithms</li></ul>Contributed<br /><ul><li>[Workflow 1]
  63. 63. [Workflow 2]
  64. 64. [Workflow 3]
  65. 65. [Workflow 4]
  66. 66. [Workflow 5]
  67. 67. …</li></ul>Community<br /><ul><li>GeoBrain
  68. 68. SciFlo
  69. 69. Data Mining
  70. 70. Giovanni</li></ul>Personal<br /><ul><li>[Workflow 1]
  71. 71. [Workflow 2]
  72. 72. [Workflow 3]
  73. 73. …</li></li></ul><li>7/27/11<br />9<br />IGARSS 2011, Vancouver, Canada<br />Laboratory Notebook<br /><ul><li>Discovery
  74. 74. Social
  75. 75. Sharing
  76. 76. Tagging
  77. 77. Discussion
  78. 78. Configuration Management
  79. 79. Versioning</li></ul>Packager<br /><ul><li>Project Manager
  80. 80. Experiment manager
  81. 81. Notebook editor</li></ul>Provisioned<br /><ul><li>Tutorials
  82. 82. User guides
  83. 83. Example uses
  84. 84. Educational packages</li></ul>Project<br /><ul><li>[Project 1]
  85. 85. [Project 2]
  86. 86. [Project 3]
  87. 87. [Project 4]
  88. 88. [Project 5]
  89. 89. …</li></ul>Community<br /><ul><li>Project results
  90. 90. Publications
  91. 91. Example cases
  92. 92. Educational packages</li></ul>Personal<br /><ul><li>Notes
  93. 93. Journals
  94. 94. …</li></li></ul><li>7/27/11<br />10<br />IGARSS 2011, Vancouver, Canada<br />Mediator<br /><ul><li>Mediates tool interaction with data
  95. 95. OPeNDAP – a common data model (accessible by most tools)
  96. 96. Custom modules reformat data for the rest of the tools
  97. 97. Ontology matches tools with data, and vice versa.</li></li></ul><li>CyberinfrastructureServices used by all other components<br />Security<br />authentication<br />authorization<br />code audit/padded cell <br />integrity checking<br />Social<br />tagging<br />sharing<br />discussions<br />groups<br />Cloud<br />elastic provisioned storage and computing<br />Discovery<br />data, tools, workflows, experiments<br />search by keyword, variable, time, author<br />Information Management<br />provenance<br />identifiers<br />archive<br />Semantic Web<br />data ontology<br />tools ontology<br />7/27/11<br />11<br />IGARSS 2011, Vancouver, Canada<br />
  98. 98. Key Advantages of ESC<br />Tool availability will be a force multiplier<br />More tools will be usable with more datasets<br />More tools will be more available to more users<br />Knowledge sharing evolves from text on paper to a rich mixture of data, tools, workflows and articles<br />A “wikihow” for Earth Science data analysis <br />Incorporating live data, services and workflows<br />ESC maintains a record of the analysis process<br />Share, repeat, build upon analysis techniques<br />Transparency of the process is built in<br />7/27/11<br />12<br />IGARSS 2011, Vancouver, Canada<br />
  99. 99. Prior Art<br />Talkoot, myExperiment.org– workflow sharing, virtual notebooks<br />Earth System Grid – provisioned tools, format standards/checkers<br />NASA Earth Exchange (NEX)<br />Land Information System – OPeNDAP as access infrastructure<br />Earth Science Modeling Framework – programmatic approach to integration<br />Giovanni, LAS – community services/tools<br />Canadian Space Science Data Portal (EOS, Feb. 22, 2011)<br />Nebula – cloud provisioning<br />7/27/11<br />13<br />IGARSS 2011, Vancouver, Canada<br />
  100. 100. A Use CaseGPM Precipitation Retrieval Algorithm Development<br />7/27/11<br />14<br />IGARSS 2011, Vancouver, Canada<br />GPM Core Satellite: Dual-Frequency Precipitation Radar (JAXA) and GPM Microwave Imager (NASA)<br />GPM Constellation: International partner satellites with mostly microwave radiometers<br />Retrieval algorithms – 3 types<br />Radar-only<br />Radiometer-only<br />Radar-radiometer-combined<br />Participants in algorithm development are distributed in Japan, NASA centers (GSFC, MSFC, JPL), NCAR, and universities (FSU, Uwisc, etc.)<br />
  101. 101. A Use CaseGPM Algorithm Development – Current Situation<br />7/27/11<br />15<br />IGARSS 2011, Vancouver, Canada<br />Interdependence among 3 types of algorithms<br />Communication/Coordination– Narrow bandwidth<br />Periodic workshop meetings and teleconferences<br />Data access – Duplicative<br />Each location/group has a copy or subset of required data<br />Sharing of data/tools – Individual, not concerted<br />through ftp/email<br />Knowledge sharing – Delayed<br />
  102. 102. A Use CaseGPM Algorithm Development – with ESC<br />7/27/11<br />16<br />IGARSS 2011, Vancouver, Canada<br />Cloud<br />Tools<br />ESC Client<br />A<br />Tools<br />ESC Client<br />Z<br />VM Image<br />VM Image<br />A<br />B<br />Tools<br />Data<br />Tools<br />Data<br />mySci Cat.<br />mySci Cat.<br />Data<br />Data<br />Community Catalog<br />ESC<br />
  103. 103. A Use CaseGPM Algorithm Development – Multi-level Membership<br />D<br />C<br />B<br />A<br />K<br />J<br />I<br />H<br />G<br />F<br />E<br />M<br />L<br />GPM<br />Combined Algorithm<br />Radar-Only<br />Radiometer-Only<br />
  104. 104. A Use CaseGPM Algorithm Development – in ESC<br />7/27/11<br />18<br />IGARSS 2011, Vancouver, Canada<br />Enhanced communication/coordination – wide bandwidth<br />Efficient data access – less duplication<br />Improved sharing – more pervasive<br />Effective knowledge sharing – immediate<br />
  105. 105. Thank you!<br />7/27/11<br />19<br />IGARSS 2011, Vancouver, Canada<br />
  106. 106. Why now?<br />Because we can do it (finally)!<br />Advances in standards acceptance andimplementation (OPeNDAP, autoconf)<br />A consistent, loosely coupled architecture encapsulates complexity and maximizes flexibility<br />Social networking has reached the mainstream<br />Key lessons can be learned from prior efforts<br />The need is growing<br />Interest in working with multiple datasets is growing<br />Calls for transparency and reproducibility are growing<br />7/27/11<br />20<br />IGARSS 2011, Vancouver, Canada<br />
  107. 107. What’s New?<br />Macro View (forest-level)<br />Systematic approach to making data available to services and vice versa<br />Integration of all major analysis components<br />Consistent view of all architectural components<br />Cyberinfrastructure services for all architectural components<br />Micro View (tree-level): Nothing!<br />7/27/11<br />21<br />IGARSS 2011, Vancouver, Canada<br />
  108. 108. How to move forward?<br />Option 1<br />RFC to community on feasibility, challenges, approach<br />Followed by RFPs for component and integration<br />Option 2<br />Narrow end-to-end prototype<br />Followed by refactoring and broadening<br />7/27/11<br />22<br />IGARSS 2011, Vancouver, Canada<br />

×