Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

  • Be the first to comment

  • Be the first to like this


  1. 1. Apache AiravataBy Shameera Rathnayaka
  2. 2. About me MSc Graduate student @ Indiana University Developer of SciGaP Lab, Indiana University Core Developer of Apache Airavata Committer and Project Management Committee Member of Apache Software Foundation (Apache Airavata, Apache Axis2) Contributor of Apache Karaf, Apache Sandesha, Apache Rampart. GSoC Student ( 2012 , 2013) GSoC Mentor 2015
  3. 3. Goals for Module Understand Apache Airavata design and implementation See how we have incorporated lessons learned for running production services. See how Apache Airavata can be extended.
  4. 4. SEAGrid, PGA and Apache Airavata SEAGrid and the PGA are clients to Apache Airavata middleware services. They run separately from Apache Airavata The PGA is a reference implementation for the Airavata API Using Airavata APIs, we can integrate other gateways PGA clones Jupyter notebooks Your favorite web framework. We’ll see how to clone the PGA in the next section
  5. 5. What is Apache Airavata? An open source, openly governed software framework for executing and managing computational jobs and workflows. Managing the metadata associated with these jobs. Supports local cluster, supercomputers, national grids, academic and commercial clouds. Basis of persistent gateway services platform (SciGaP)
  6. 6. Airavata Architectural Goals ... Distributed Systems Concepts Scalability Fault Tolerance Security Component-Based Architecture Loosely Coupled Components Extension and expansion points Operational Experience Fault Handling: user, resource, and other errors Experiment Recovery Reliable Job Monitoring
  7. 7. High Level Architecture AMQP Messaging API server Work Queue Worker Orchestrator Computational Resources Job Monitor Registry Workflow Engine API Server User
  8. 8. Support Multiple Gateways AMQP Messaging API server Work Queue Worker Orchestrator Computational Resources Job Monitor Registry Workflow Engine API Server Gateway B Gateway C Gateway AUser User User
  9. 9. Why Component Based Architecture Pattern? Each component has specific work to do. API Server – Hide all component from User Orchestrator – Take Decisions and Selection Worker – Execute set of Tasks Registry - Data Catalog Workflow Engine – Workflow Enactment Easy to evolve with new technologies. AMQP messaging provides inter-component communications. You can add new components as subscribers to system messages
  10. 10. Scalability , Fault-tolerance and Recoverability Airavata worker capacity can be increased and decreased on demand to maintain performance and load spikes. Airavata Workers scale horizontally. Distribute jobs between workers using the internal work queue. Worker Worker Work Queue Orchestrator
  11. 11. Operational Fault Handling User Errors Pluggable job validation rules Data Staging Errors Retry on failure depend on situation ex: network glitches Job Submission Errors Retry on failure depend on situation ex: ssh connection issue, queue limit Inform administrator in allocation issues Verify Job submission Job Failures on Remote Compute Resource Copy standard out and error files
  12. 12. Security It is important to have user authentication and authorization. Airavata API security with WSO2 Identity Server. Credential store manages all machine credentials. SSH keys SSH username & passwords. Airavata provide user permission based on security role. Super administrator Administrator User
  13. 13. Other Features …
  14. 14. Other Features … Apache Thrift-based API allows users to use whatever language they prefer in client side. Or even you can plug different backend components written in different languages. Extendibility with multiple extension points. Don’t require large infrastructure. Incremental update with almost zero downtime. Minimum overhead to dev ops.
  15. 15. Multiple Job Monitoring … Airavata default job monitoring method is email Airavata has SSH base job monitoring support too Airavata UNICORE job monitoring Plug your own job monitoring mechanism
  16. 16. Easy to plug new compute resources Airavata comes with SLURM and PBS base job submission. Airavata template mechanism make easy to plug different type of Job submissions types. Airavata has been extended to submit jobs to Jureca supercomputer which has web service interface. Integrating Jetstream as new computer resource and Jetstream cloud provisioning.
  17. 17. How to get your client to talk with Airavata? Airavata provide multiple client sdks. We have heavily working with php (PGA web client), python (Jupyter and dev test) and java (desktop client). Use Airavata provided client sdk to talk directly to API Server. Apache Airavata Apache Airavata
  18. 18. Road Map Highlight … Airavata Data Management Organization, Analystics, Collaboration Airavata data analytic tool Airavata workflow support Airavata OpenStack support Ansible scripts to automate large scale deployment in one click Airavata component Docker containers Airavata Mesos integration
  19. 19. Questions ?
  20. 20. Additional Resources
  21. 21. Experiment Execution Worker Worker Computational Resources Email Notification Submit Job Submit Job API server Experiment API Server Registry Apache Zookeeper Work Queue Orchestrator
  22. 22. Execution Models Experiment Model Process Model Task Model Job Model Task Model Task Model Task Model Process ModelProcess ModelProcess Model Task ModelTask ModelTask Model
  23. 23. Execution Flow