Apache Airavata is a system that allows scientists to automate computational experiments and workflows without manual intervention. It collects experiment data and parameters, executes applications and workflows on computational resources, and returns results while providing ongoing progress updates to the user. Airavata has four main components - a workflow interpreter to manage execution, a resource manager to control applications and data transfers, a registry to define available applications and store results, and a messaging system to communicate progress.
Automating Deployments with Bamboo and Ansible - Randall Thomson, Senior TechOps Engineer - LogicMonitor
LogicMonitor uses Atlassian Bamboo and Ansible to manage the deployment of applications throughout their microservice based infrastructure. The process integrates tightly with the LogicMonitor API to programmatically set SDTs and OpsNotes. Additional integration with HipChat sends automated room notifications. Randall Thomson will speak on how the LogicMonitor TechOps team uses Ansible and Bamboo to empower their Development team to safely and securely deploy applications in test and production environments.
AI Powered Full Stack Monitoring using Dynatrace - Himanshu Chhetri, CTO - Addteq
How do you effectively monitor the health of your Atlassian ecosystem and easily troubleshoot issues? DynaTrace, one of the recommended monitoring tools in Atlassian's enterprise documentation, is capable of automatically detecting performance issues in infrastructure, application and even provide insights into user experience across the globe. Himanshu Chhetri will present insights and real-world use cases using DynaTrace to monitor your mission-critical Atlassian tools.
In this webinar, we will take a look at the Azure Machine Learning Studio and explore the features that it has to offer. We will take a look on how to create a predictive analytic solution and discuss how to deploy the solution as a web service. This will include a case study on Flight Delay Prediction Analysis with Power BI as well.
Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM.
This slides is for my lecture in #zconf 2016. I introduced actor model and some features provided in Akka.
The goal of this talk is to highlight open source opportunities for students especially through an opportunity to earn $5000 through Google Summer of Code program. I will discuss some of the tips on how to engage with open source communities, the befits for contributing. I will provide motivating examples on how students can gain significant experience in contributing challenging distributed systems problems while impacting scientific research. I will specifically focus with a concrete example of Apache Airavata software suite for Web-based science gateways. I will list some example GSoC topics of interest and provide some recipes for success in getting accepted and navigating through success.
Automating Deployments with Bamboo and Ansible - Randall Thomson, Senior TechOps Engineer - LogicMonitor
LogicMonitor uses Atlassian Bamboo and Ansible to manage the deployment of applications throughout their microservice based infrastructure. The process integrates tightly with the LogicMonitor API to programmatically set SDTs and OpsNotes. Additional integration with HipChat sends automated room notifications. Randall Thomson will speak on how the LogicMonitor TechOps team uses Ansible and Bamboo to empower their Development team to safely and securely deploy applications in test and production environments.
AI Powered Full Stack Monitoring using Dynatrace - Himanshu Chhetri, CTO - Addteq
How do you effectively monitor the health of your Atlassian ecosystem and easily troubleshoot issues? DynaTrace, one of the recommended monitoring tools in Atlassian's enterprise documentation, is capable of automatically detecting performance issues in infrastructure, application and even provide insights into user experience across the globe. Himanshu Chhetri will present insights and real-world use cases using DynaTrace to monitor your mission-critical Atlassian tools.
In this webinar, we will take a look at the Azure Machine Learning Studio and explore the features that it has to offer. We will take a look on how to create a predictive analytic solution and discuss how to deploy the solution as a web service. This will include a case study on Flight Delay Prediction Analysis with Power BI as well.
Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM.
This slides is for my lecture in #zconf 2016. I introduced actor model and some features provided in Akka.
The goal of this talk is to highlight open source opportunities for students especially through an opportunity to earn $5000 through Google Summer of Code program. I will discuss some of the tips on how to engage with open source communities, the befits for contributing. I will provide motivating examples on how students can gain significant experience in contributing challenging distributed systems problems while impacting scientific research. I will specifically focus with a concrete example of Apache Airavata software suite for Web-based science gateways. I will list some example GSoC topics of interest and provide some recipes for success in getting accepted and navigating through success.
Apache Apex brings you the power to quickly build and run big data batch and stream processing applications. But what about visualizing your data in real time as it flows through the Apache Apex applications? Together, we will review Apache Apex, and how it integrates with Apache Hadoop and Apache Kafka to process your big data with streaming computation. Then we will explore the options available to visualize Apex applications metrics and data, including open-source options like REST and PubSub mechanisms in StrAM, as well as features available in the RTS Console like real-time Dashboards and Widgets. We will also look into ways of packaging dashboards inside your Apache Apex applications.
This presentation gives an overview of the Apache Airavata project. It explains Apache Airavata in terms of it's architecture, data models and user interface.
Links for further information and connecting
http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
https://nz.linkedin.com/pub/mike-frampton/20/630/385
https://open-source-systems.blogspot.com/
In this tutorial, we will build a demo application to learn and implement Angular Resolver. Clone the github repository and start playing with the code.
StackWatch: A prototype CloudWatch service for CloudStackChiradeep Vittal
Presented at CloudStack Collab 2014 in Denver. The presentation explores adding a Cloudwatch service to Apache CloudStack and some of the interesting design decisions and consequences.
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...Provectus
Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing pipelines, and also data ingestion and integration flows, supporting for both batch and streaming use cases. In presentation I will provide a general overview of Apache Beam and programming model comparison Apache Beam vs Apache Spark.
Sherlock Homepage - A detective story about running large web services (VISUG...Maarten Balliauw
The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
Cyberinfrastructure Experiences with Apache Airavatasmarru
In this short presentation, we summarize the Apache Airavata's use of component-based architecture to encompass major gateway capabilities (such as metadata management, meta-scheduling, execution management, and messaging).
Apache Apex brings you the power to quickly build and run big data batch and stream processing applications. But what about visualizing your data in real time as it flows through the Apache Apex applications? Together, we will review Apache Apex, and how it integrates with Apache Hadoop and Apache Kafka to process your big data with streaming computation. Then we will explore the options available to visualize Apex applications metrics and data, including open-source options like REST and PubSub mechanisms in StrAM, as well as features available in the RTS Console like real-time Dashboards and Widgets. We will also look into ways of packaging dashboards inside your Apache Apex applications.
This presentation gives an overview of the Apache Airavata project. It explains Apache Airavata in terms of it's architecture, data models and user interface.
Links for further information and connecting
http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
https://nz.linkedin.com/pub/mike-frampton/20/630/385
https://open-source-systems.blogspot.com/
In this tutorial, we will build a demo application to learn and implement Angular Resolver. Clone the github repository and start playing with the code.
StackWatch: A prototype CloudWatch service for CloudStackChiradeep Vittal
Presented at CloudStack Collab 2014 in Denver. The presentation explores adding a Cloudwatch service to Apache CloudStack and some of the interesting design decisions and consequences.
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...Provectus
Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing pipelines, and also data ingestion and integration flows, supporting for both batch and streaming use cases. In presentation I will provide a general overview of Apache Beam and programming model comparison Apache Beam vs Apache Spark.
Sherlock Homepage - A detective story about running large web services (VISUG...Maarten Balliauw
The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
Cyberinfrastructure Experiences with Apache Airavatasmarru
In this short presentation, we summarize the Apache Airavata's use of component-based architecture to encompass major gateway capabilities (such as metadata management, meta-scheduling, execution management, and messaging).
RESTLess Design with Apache Thrift: Experiences from Apache Airavatasmarru
Apache Airavata is software for providing services to manage scientific applications on a wide range of remote computing resources. Airavata can be used by both individual scientists to run scientific workflows as well as communities of scientists through Web browser interfaces. It is a challenge to bring all of Airavata’s capabilities together in the single API layer that is our prerequisite for a 1.0 release. To support our diverse use cases, we have developed a rich data model and messaging format that we need to expose to client developers using many programming languages. We do not believe this is a good match for REST style services. In this presentation, we present our use and evaluation of Apache Thrift as an interface and data model definition tool, its use internally in Airavata, and its use to deliver and distribute client development kits.
The success of the Google Summer of Code program within ASF demonstrates the interest and potential impact Apache projects could have on grooming next generation software developers. Many projects have benefited from the GSoC contributions and some have succeeded in retaining the students as active PMC members. While GSoC is a good vehicle for potential student committers, we could extend the impact and broaden the reach. Beyond GSoC, currently there is no compelling mechanism for interested students to venture into the 150+ Apache project issue trackers to find out an interesting topic to contribute. We propose to build on the GSoC success and create a common forum for PMC’s to propose topics and volunteer to mentor well defined and suitably scoped student research projects. These student projects create a win-win situation for both the Apache projects and the students.
As an exemplar, we will discuss the Apache Airavata project engagement with student academic projects. The globally distributed locations of PMC members of the Apache Airavata project has resulted in the successful launch of many student research projects in the US, Indian and Sri Lanka. Brief descriptions of the projects, their inclusion within existing university curricula and their successes and challenges will be presented. We will then elaborate on how these experiences can be generalized and modeled as a systematic mechanism to catalyze student research projects. While particularly sharing the experiences from developing countries, we discuss how these ideas are globally applicable in exposing students to the ASF model, enabling them to discuss their ideas and work with leading researchers and open source developers around the world, motivating them through virtual hackathons and eventually creating potential pathways to Apache Committership.
The proposed effort raises many open questions. However, initiated through this talk, we would like to hear feedback from Apache projects and the user community and take the idea further with the Apache Community Development PMC.
This talk introduces the Apache Airavata software for executing and managing computational jobs on distributed computing resources including local clusters, supercomputers, national grids, academic and commercial clouds. Airavata is currently used to build Web-based science gateways and assist to compose, manage, execute, and monitor large scale applications and workflows composed of these services.
8. How can I make this
much simpler…?
Logically, this is how
my life would be
made easier…
Is it possible to
automate this flow
sequence without my
guidance?
9. Scientists from many
different fields face this
problem everyday.
The solution is to use a
workflow-powered
science gateway to
manage the experiment
online.
What is a workflow you
ask?
Well, you just saw one in
our previous animation…
10. We introduce Apache Airavata, a system capable of
composing, managing, executing, and monitoring
small to large scale applications and workflows
Want to see how it works?
A Typical Workflow
11. Apache Airavata
I will handover my data & my
experiment details (the workflow)
to the Airavata server
The Gateway
Airavata will complete the
experiment & return me the results
Results
Progress of the experiment
… and while I wait for
results, Airavata will notify me with
progress updates of my experiment
12. Let’s look closely how Airavata
manages workflows.
The GatewayResults
Experiment progress
Apache Airavata
13. Let’s look closely how Airavata
manages workflows.
The GatewayResults
Experiment progress
14. Airavata main has 4 components…
The Gateway
1. Workflow Interpreter
Steer the workflow execution
2. The GFac
Steer science app executions & data
transfers
Workflow Interpreter
GFac
Message Box
Registry
3. The Registry
Defines the available applications &
records all results of experiments
4. The Message Box
Records the progress of the workflow
execution
15. A Stable API for
Airavata
Apache Airavata
EndUsersGatewayDeveloper
Scientific
Application
Computational Resources
16. Application
Registration
UI
Application
Developer
A1
Airavata Service
Interface
(wraps client API)
Service Map
XML
Airavata
Server
Service Map
to AWSDL
Web Based
workflow
composerWorkflow
Developer
Get AWSDL
Put XWF
A2
A3
W1
W2
W3
Web Based
Experiment
BuilderExperiment
Builder
Get WI’s
E1
E2
E3
W4
Shred
Workflow
Inputs
Launch
Workflow
Web Based
Workflow
MonitorWatch Progress
M1
M2
Get Workflow
Graph
M3
Monitor
Workflow
17. Goal of the project
• Design Web-Based interfaces for Airavata:
– Application Registration
– Workflow Construction
– Workflow Execution
– Workflow Monitoring
• Provide an opportunity for GSoC to
understand Distributed System in action
• Scope for Research and Software Engineering
papers
18. Data Model
• Application Description
– User describes inputs and outputs of the
application.
– Currently this information is captured in Service
Map Schema.
– This schema is stored in Airavata Registry as XML.
Also the schema utility generates a application
service WSDL from this schema using the Airavata
WSDL Generator.
19. Application
Registration
UI
Application
Developer
A1
Service Map
XML
Service Map
to AWSDL
Web Based
workflow
composer
Workflow
Developer
Get AWSDL
A2
W1
W2
XML
AiravataServerAPI
Launch & Manage Jobs
Notify progress of job
or workflow execution
Real-Time
Monitoring
Messaging
Subsystem
Registry
Execute & Manage
Computations
Workflow
Interpreter
Application
Factory (Gfac)
Applicatio
n Desc
Applicatio
n Desc
Workflow
20. A peek at one of the cluster
Interconnect
Nodes
21. Scheduling ‘qsub’ batch jobs on the cluster
worker
node
worker
node
worker
node
worker
node
worker
node
SGE MASTER node
Queue-A Queue-B Queue-C
ASlot1
ASlot2
BSlot1
CSlot1
CSlot2
CSlot3
BSlot1
BSlot2
BSlot3
BSlot1
CSlot1
CSlot2
ASlot1
BSlot1
CSlot1
Queues
Policies
Priorities
Share/Tickets
Resources
Users/Projects
JOB Y JOB Z
JOB X
JOB U
JOB O
JOB N
