A Look into the Apache OODT Ecosystem


Published on

An overview of the Apache OODT ecosystem projects and the work going on in the space. How we got to Apache. Other good stuff.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A Look into the Apache OODT Ecosystem

  1. 1. A Look into the Apache OODT Ecosystem Chris A. Mattmann NASA JPL/Univ. Southern California/ASF [email_address] November 9, 2011
  2. 2. <ul><li>Apache Member involved in </li></ul><ul><ul><li>OODT (VP, PMC), Tika (VP,PMC), Nutch (PMC), Incubator (PMC), SIS (Mentor), Lucy (Mentor) and Gora (Champion), MRUnit (Mentor), Airavata (Mentor) </li></ul></ul><ul><li>Senior Computer Scientist at NASA JPL in Pasadena, CA USA </li></ul><ul><li>Software Architecture/Engineering Prof at Univ. of Southern California </li></ul>And you are?
  3. 3. Welcome to the Apache in Space! (OODT) Track
  4. 4. Agenda <ul><li>Overview of OODT and its history </li></ul><ul><li>How we got it to Apache </li></ul><ul><li>How other projects can follow our model </li></ul><ul><li>Existing successful deployments of OODT </li></ul><ul><li>Pointers to papers, and more information including case studies </li></ul>
  5. 5. Lessons from 90’s era missions <ul><li>Increasing data volumes (exponential growth) </li></ul><ul><li>Increasing complexity of instruments and algorithms </li></ul><ul><li>Increasing availability of proxy/sim/ancillary data </li></ul><ul><li>Increasing rate of technology refresh </li></ul><ul><li>… all of this while NASA Earth Mission funding was decreasing </li></ul>A data system framework based on a standard architecture and reusable software components for supporting all future missions.
  6. 6. Enter OODT Object Oriented Data Technology http://oodt.apache.org Funded initially in 1998 by NASA ’s Office of Space Science Envisaged as a national software framework for sharing data across heterogeneous, distributed data repositories OODT is both an architecture and a reference implementation providing Data Production Data Distribution Data Discovery Data Access OODT is Open Source and available from the Apache Software Foundation
  7. 7. Apache OODT <ul><li>Originally funded by NASA to focus on </li></ul><ul><ul><li>distributed science data system environments </li></ul></ul><ul><ul><li>science data generation </li></ul></ul><ul><ul><li>data capture, end-to-end </li></ul></ul><ul><ul><li>Distributed access to science data repositories by the community </li></ul></ul><ul><li>A set of building blocks/services to exploit common system patterns for reuse </li></ul><ul><li>Supports deployment based on a rich information model </li></ul><ul><li>Selected as a top level Apache Software Foundation project in January 2011 </li></ul><ul><li>Runner up for NASA Software of the Year </li></ul><ul><li>Used for a number of science data system activities in planetary, earth, biomedicine, astrophysics </li></ul>http://oodt.apache.org
  8. 8. Apache OODT Press
  9. 9. Why Apache and OODT? <ul><li>OODT is meant to be a set of tools to help build data systems </li></ul><ul><ul><li>It ’s not meant to be “turn key” </li></ul></ul><ul><ul><li>It attempts to exploit the boundary between bringing in capability vs. being overly rigid in science </li></ul></ul><ul><ul><li>Each discipline/project extends </li></ul></ul><ul><li>Apache is the elite open source community for software developers </li></ul><ul><ul><li>Less than 100 projects have been promoted to top level (Apache Web Server, Tomcat, Solr, Hadoop) </li></ul></ul><ul><ul><li>Differs from other open source communities; it provides a governance and management structure </li></ul></ul>
  10. 10. Governance Model+NASA=&hearts; <ul><li>NASA and other government agencies have tons of process </li></ul><ul><ul><li>They like that </li></ul></ul>
  11. 11. Publicly accessible and searchable archives <ul><li>http://svnsearch.org/svnsearch/repos/ASF/search?path=%2Foodt </li></ul><ul><li>http://mail-archives.apache.org/mod_mbox/oodt-dev/ </li></ul><ul><li>http://mail-archives.apache.org/mod_mbox/oodt-user/ </li></ul><ul><li>100+ ML list subscriptions </li></ul>
  12. 12. Great Metrics and Insight <ul><li>http://www.ohloh.net/p/oodt </li></ul>
  13. 13. Movement to the ASF <ul><li>Meeting held June 15, 2007 at JPL with ASF President Justin Erenkrantz </li></ul><ul><ul><li>Develop plan moving forward to bring first NASA project to Apache </li></ul></ul><ul><ul><li>Discuss obstacles, sponsorship </li></ul></ul><ul><ul><li>Discuss outlook </li></ul></ul>
  14. 14. 2007: original goals <ul><li>Come up with incubation proposal </li></ul><ul><ul><li>Chris Mattmann was one of the principal contributors to the proposal for the Tika project, and to other Incubation activities (Apache SIS) </li></ul></ul><ul><ul><li>Send out emails to the Incubator mailing list </li></ul></ul><ul><ul><ul><li>Look for mentors </li></ul></ul></ul><ul><li>Get sponsorship from ranking Apache PMC member or board member </li></ul><ul><ul><li>Justin and others </li></ul></ul><ul><li>Top-level project versus sub project outlook heading out of incubation </li></ul>
  15. 15. OODT Incubator Planning <ul><li>Monthly Updates (for first 3 months, then quarterly) </li></ul><ul><ul><li>Status </li></ul></ul><ul><ul><li>Progress </li></ul></ul><ul><ul><li>Community </li></ul></ul><ul><ul><li>Acceptance </li></ul></ul><ul><li>Plan for exiting incubation </li></ul><ul><ul><li>How to have a solid user base </li></ul></ul><ul><ul><li>How to operate as a unit in the Apache way </li></ul></ul><ul><ul><li>Maintenance of user interest and community going forward </li></ul></ul>
  16. 16. OODT’s next steps circa 2007 <ul><li>JPL to tackle legal issues </li></ul><ul><ul><li>Is OODT releasable as an Apache product </li></ul></ul><ul><ul><li>http://www.apache.org/licenses/software-grant.txt </li></ul></ul><ul><ul><ul><li>This needs to be signed by parties that be by JPL </li></ul></ul></ul><ul><ul><li>Contributor License Agreement </li></ul></ul><ul><ul><ul><li>Do we need a corporate one? </li></ul></ul></ul><ul><li>In parallel to this </li></ul><ul><ul><li>Draft OODT incubation proposal </li></ul></ul><ul><ul><li>Start identifying who would initially be interested </li></ul></ul><ul><ul><ul><li>More external, non-JPL people who are interested, the better </li></ul></ul></ul><ul><li>Justin to get slides from other incubator people </li></ul>
  17. 17. … 2 years later <ul><li>Worked it out with JPL legal </li></ul><ul><ul><li>Turns out the ALv2 license is extremely friendly and is something that JPL (note not all of NASA) was amenable to </li></ul></ul><ul><li>Developed OODT incubator proposal </li></ul><ul><ul><li>http://wiki.apache.org/incubator/OODTProposal </li></ul></ul><ul><li>Found willing Apache mentors besides Justin </li></ul><ul><ul><li>Jean-Frederic Clere, Ross Gardler, Ian Holsman </li></ul></ul><ul><li>… Put OODT at Apache! </li></ul>
  18. 18. Apache OODT Community <ul><li>Includes PMC members from </li></ul><ul><ul><li>NASA JPL, Univ. of Southern California, Google, Children’s Hospital Los Angeles (CHLA), Vdio, South African SKA Project </li></ul></ul><ul><li>Projects that are deploying it operationally at </li></ul><ul><ul><li>Decadal-survey recommended NASA Earth science missions, NIH, and NCI, CHLA, USC, South African SKA project </li></ul></ul><ul><li>Use in the classroom </li></ul><ul><ul><li>My graduate-level software architecture and seach engines courses </li></ul></ul>
  19. 19. OODT Framework OBJECT ORIENTED DATA TECHNOLOGY FRAMEWORK OODT/Science Web Tools Archive Client Profile XML Data Data System 1 Data System 2 Catalog & Archive Service Profile Service Product Service Query Service Bridge to External Services Navigation Service Other Service 1 Other Service 2 Andrew Hart and Emily Law will talk about these later You’ll hear about this later today I’ll tell you about these now
  20. 20. Architectural Principles <ul><li>Division of Labor </li></ul><ul><ul><li>Don ’ t make one component the workhorse! </li></ul></ul><ul><li>Technology Independence </li></ul><ul><ul><li>Don ’ t get bitten in the rear when a software vendor decides to charge you a lot of $$$ for their previously low cost technology </li></ul></ul><ul><li>Metadata as a first-class citizen </li></ul><ul><ul><li>Descriptions of resources come in handy </li></ul></ul><ul><li>Separation of software and data models </li></ul><ul><ul><li>Allow each to evolve independently </li></ul></ul>
  21. 21. OODT Architecture <ul><li>Reference Architecture </li></ul><ul><ul><li>Four pairs of component types </li></ul></ul><ul><ul><ul><li>Product Client/Server, Profile Client/Server, Query Client/Server, Catalog and Archive Client/Server </li></ul></ul></ul><ul><ul><li>Two connector types </li></ul></ul><ul><ul><ul><li>Messaging layer discussed in http://sunset.usc.edu/~mattmann/pubs/ICSE.pdf </li></ul></ul></ul><ul><ul><ul><li>Handler connector (discussed in this presentation) </li></ul></ul></ul><ul><li>Instantiated for different domains using these fundamental building blocks </li></ul>
  22. 22. Product Client and Server Product Client (A) Product Server (A) RAID Disk <ul><li>Deliver data from underlying data store </li></ul><ul><li>Accept uniform query structure that identifies 0 or more “ products ” (data items) to retrieve </li></ul><ul><li>Many-to-Many </li></ul>Web site MSSQL Product Client (B) Product Client (C) Product Server (B) Product Server (C)
  23. 23. How about an example of a product?
  24. 24. Profile Client and Server Profile Client (A) Profile Server (A) Oracle <ul><li>Deliver metadata from underlying metadata store </li></ul><ul><li>Metadata gives user enough information about where to find actual data </li></ul><ul><ul><li>Housekeeping information </li></ul></ul><ul><ul><li>Resource information </li></ul></ul><ul><ul><li>Domain-specific information </li></ul></ul><ul><li>Many-to-Many </li></ul>Web site MSSQL Profile Client (B) Profile Client (C) Profile Server (B) Profile Server (C)
  25. 25. How about an example of a profile? Attributes Relationships Credit: A. Hart
  26. 26. Query Client and Server <ul><li>Query Server seeded with initial set of pointers to Profile Servers </li></ul><ul><li>Profile Servers point to actual resources (Product Servers, even other Profile Servers ) </li></ul><ul><li>Interactive (metadata returned) and non-interactive (data returned) </li></ul><ul><li>Many-to-Many </li></ul>Query Server (A) Query Client (A) Product Server (A) Product Server (B) Discovered Profile Server (B) Profile Server (A) Initial set Query Client (B) Query Client (C) Query Server (B) Query Server (C)
  27. 27. Catalog and Archive Client and Server (CAS) <ul><li>Ingest data into repository and metadata into registry </li></ul><ul><li>Run processing algorithms on data/metadata upon ingestion </li></ul><ul><li>Workflow support </li></ul><ul><li>Serve back Repository data with Product Server </li></ul><ul><li>Serve back Registry metadata with Profile Server </li></ul><ul><li>Many-to-Many </li></ul>Archive Client (A) Archive Server (A) Repository Registry Profile Server (A) Product Server (A) Archive Server (B) Archive Client (B) Archive Client (C)
  28. 28. Some notes about CAS <ul><li>All Core components implemented as web services </li></ul><ul><ul><li>XML-RPC used to communicate between components </li></ul></ul><ul><ul><li>Servers implemented in Java </li></ul></ul><ul><ul><li>Clients implemented in Java, scripts, Python, PHP and web-apps </li></ul></ul><ul><ul><li>Service configuration implemented in ASCII and XML files </li></ul></ul>Credit: D. Woollard
  29. 29. Handler Connectors RAID Disk <ul><li>Encapsulate (meta-)data coordination and communication </li></ul><ul><li>Allow for dynamic addition and removal of different classes of back end metadata and data stores </li></ul>DBMS Product Handler Flat File Product Handler Web Site Product Handler Product/Profile Server Product/Profile Server Web site MSSQL MSSQL
  30. 30. Example handler connectors <ul><li>XMLPS </li></ul><ul><ul><li>http://oodt.apache.org/components/maven/xmlps/ </li></ul></ul><ul><ul><li>XML config file specifies recipe for extracting records from an RDBMS and turning them into a NoSQL repository </li></ul></ul><ul><li>PS </li></ul><ul><ul><li>XML configurable profile server to unlock OPeNDAP datasets and pass them to OODT </li></ul></ul>
  31. 31. So, how do you piece them together: NASA VODC <ul><li>NASA’s Virtual Oceanographic Data Center (VODC) </li></ul><ul><li>http://vodc.jpl.nasa.gov </li></ul><ul><li>Information integration using OODT components </li></ul><ul><ul><li>Profile, Product, Query, also uses Apache Solr, and Plone </li></ul></ul>
  32. 32. So, how do you piece them together: JPL’s CDX <ul><li>CDX = Climate Data Exchange </li></ul><ul><li>Provide comparison of remote sensing data and model outputs </li></ul><ul><li>Existing systems remain in place; services expose data and functions over the network; support the era of IPCC 5 th assessment and distributed, petabytes of data </li></ul>
  33. 33. Who’s doing what? <ul><li>Children’s Hospital Los Angeles </li></ul><ul><ul><li>Improving upon XMLPS, and CAS (Andrew Hart + Ricky Nguyen will talk about this) </li></ul></ul><ul><ul><li>Supporting data analytics </li></ul></ul><ul><li>Google </li></ul><ul><ul><li>Brian Foster working on command line improvements and data protocol push/pull </li></ul></ul><ul><li>SKA South Africa </li></ul><ul><ul><li>Deploying file manager and crawler for use in KAT-7 pipeline ingestion </li></ul></ul><ul><li>NIH/NCI </li></ul><ul><ul><li>Maintaining the XMLPS components, and CAS components </li></ul></ul><ul><ul><li>Helping with user interfaces </li></ul></ul><ul><li>Various JPL and NASA research projects </li></ul><ul><ul><li>OPeNDAPps, XMLPS </li></ul></ul><ul><li>Various NASA missions </li></ul><ul><ul><li>Workflow, PCS, services, OPSui, other web apps </li></ul></ul>
  34. 34. Latest release: 0.3 <ul><li>First appearance of PCS </li></ul><ul><ul><li>Core, Services (JAX-RS) </li></ul></ul><ul><li>Web Applications </li></ul><ul><ul><li>Balance (PHP), and Wicket (Java)-based apps for file management and workflow monitoring </li></ul></ul><ul><li>First release deployed to Maven Central </li></ul><ul><ul><li>We did backport 0.2 there after this </li></ul></ul><ul><ul><li>Over 60 issues fixed in JIRA </li></ul></ul><ul><li>June 2011: recommended stable release </li></ul>
  35. 35. Working on: 0.4 <ul><li>Operator Interface (OODT-157) </li></ul><ul><ul><li>Andrew Hart and I will talk about this </li></ul></ul><ul><li>Workflow2 integration (OODT-215) and all of its sub-issues </li></ul><ul><ul><li>Global workflow conditions, dynamic workflows, parallel/sequential model, new workflow engine, etc. </li></ul></ul><ul><li>OODT RADIX for super easy deployment (OODT-120) </li></ul><ul><ul><li>Paul Ramirez and Cameron Goodale will discuss this </li></ul></ul><ul><li>Solr sync with File Manager (OODT-326) </li></ul><ul><li>Improvements to XMLPS (OODT-333) and new crawler actions (OODT-33, OODT-34, OODT-35, OODT-36, OODT-37) </li></ul><ul><li>Over 48 issues currently resolved </li></ul><ul><li>Likely to come before end of Q4 2011 </li></ul>
  36. 36. Using Apache OODT as a testbed for software process <ul><li>Missions maintain their own local CMs </li></ul><ul><li>Local mission CMs contain forks of existing OSS software </li></ul><ul><ul><li>Forks can be patch based or CM based </li></ul></ul><ul><li>Changes found particularly effective are discussed within the comm. And eventually brought before a CCB that reviews their generality, etc. </li></ul>Credit: D. Freeborn
  37. 37. Some Grand Challenges I’m interested in: OODT can help! <ul><li>How do we handle 700 TB/sec of data coming off the wire when we actually have to keep it around? </li></ul><ul><ul><li>Required by the Square Kilometre Array </li></ul></ul><ul><li>Joe scientist says I’ve got an IDL or Matlab algorithm that I will not change and I need to run it on 10 years of data from the Colorado River Basin and store and disseminate the output products </li></ul><ul><ul><li>Required by the Western Snow Hydrology project </li></ul></ul>
  38. 38. Some Grand Challenges I’m interested in: OODT can help! <ul><li>How do we compare petabytes of climate model output data in a variety of formats (HDF, NetCDF, Grib, etc.) with petabytes of remote sensing data to improve climate models for the next IPCC assessment? </li></ul><ul><ul><li>Required by the 5 th IPCC assessment and the Earth System Grid and NASA </li></ul></ul><ul><li>How do we catalog all of NASA’s current planetary science data? </li></ul>
  39. 39. Key Takeaway OODT is already doing and/or preparing the world to handle all of these diverse use cases! It’s a constantly evolving and improving framework – join up and help. It’s free and open source from Apache and helping government demonstrate the public good
  40. 40. OODT Project Contact Info <ul><li>Learn more and track our progress at: </li></ul><ul><ul><li>http://oodt.apache.org </li></ul></ul><ul><ul><li>WIKI: https://cwiki.apache.org/OODT / </li></ul></ul><ul><ul><li>JIRA: https://issues.apache.org/jira/browse/OODT </li></ul></ul><ul><li>Join the mailing list: </li></ul><ul><ul><li>[email_address] </li></ul></ul><ul><li>Chat on IRC: </li></ul><ul><ul><li>#oodt on irc.freenode.net </li></ul></ul><ul><li>Acknowledgements </li></ul><ul><ul><li>Key Members of the OODT teams: Chris Mattmann, Daniel J. Crichton, Steve Hughes, Andrew Hart, Sean Kelly, Sean Hardman, Paul Ramirez, David Woollard, Brian Foster, Dana Freeborn, Emily Law, Mike Cayanan, Luca Cinquini, Heather Kincaid </li></ul></ul><ul><ul><li>Projects, Sponsors, Collaborators: Planetary Data System, Early Detection Research Network, Climate Data Exchange, Virtual Pediatric Intensive Care Unit, NASA SMAP Mission, NASA OCO-2 Mission, NASA NPP Sounder Peate, NASA ACOS Mission, Earth System Grid Federation </li></ul></ul>
  41. 41. Alright, I ’ll shut up now <ul><li>Any questions? </li></ul><ul><li>THANK YOU! </li></ul><ul><ul><li>[email_address] </li></ul></ul><ul><ul><li>@chrismattmann on Twitter </li></ul></ul>