SlideShare a Scribd company logo
OGCE WorkflowSuite for Science
         Gateways

  Suresh Marru, Raminder Singh,
 Chathura Herath & Marlon Pierce

        Indiana University
OGCE

       Gateways      TeraGrid
                    User Portal
        (LEAD,
      GridChem,
          …)


                  TG GIG




                                   Generalize,
                                  Harden, Build
                                      Test
Gateways/E-Science Community
Requirements from gateways

• Gateways demand scientific workflow systems
  to be:
  – Flexible
  – Dynamic
  – Interactive
  – Technology Adaptive
  – Interoperable with Emerging Computational
    Resources and their job management interfaces
OGCE Workflow Suite
• Generic Service Toolkit
   –   Tool to wrap command-line applications as web services
   –   Handles file staging&job submissions
   –   Extensible runtime for security, resource brokering& urgent computing
   –   Generic Factory service for on-demand creation of application services
• XRegistry
   – Information repository for the OGCE workflow suite
   – Register, search, retrieve&share XML documents
   – User & hierarchical group based authorization
• XBaya
   – GUI based tool to compose&monitor workflows
   – Extensible support for compiler plug-ins like BPEL &Jython
   – Dynamic Workflow Execution support to start, pause, resume, rewind
     of workflow executions

                         OGCE Workflow Tutorial
Features
• Security
   –   Authentication and authorization
   –   Secure invocations between services
   –   Support for gateway community accounts
   –   Support for multiple user accounts
• Reliability
   – Retry job submissions and file staging
   – Fault Tolerance and Recovery service
        • Over-provisioning and migration
• Compatibility
   – Taverna, Kepler and Trianna


                        OGCE Workflow Tutorial
Application Services
• Workflows are built by composing web                  Application Factory

  services                                                        c

  – Fortran applications are “wrapped” by a
    Application Factory which generates a web
    service for the app.
     • Registers WSDL for the service with a registry
                                                            App
  – Each service generates a stream of                      Service
    notifications that log the service actions back
    to the XMC Cat Metadata Catalog.
                                                          Run program
                                                          & publish events
Workflow Composition, Execution
                & Monitoring
Baya enables users to
 construct, share, execute
 and monitor sequence of
 tasks executing on their
 local workstations to
 high-end compute
 resources.
Service Monitoring via Events
• The service output is a stream of events                                           Application
                                                                                       Service
    –   I am running your request                                                     Instance
    –   I have started to move your input files.
    –   I have all the files                                                6
                                                                        5
    –   I am running your application.                              4
    –   The application is finished                             3
    –   I am moving the output to you file space            2
                                                        1
    –   I am done.
• These are automatically generated
  by the service using a
  distributed event system
  (WS-Eventing / WS-Notification)                                           Notification
    – Topic based pub-sub system with                                        Channel
      a well known “channel”.
                                                   Subscribe
                                                   Topic=x                                    x
                                                                            x

                                                            listener                    publisher
OGCE Workflow Tools




WRF-Static running
  on Tungsten
Workflow Suite Architecture
XML Metadata Catalog (XMC Cat)
                         Taming Complex Scientific Metadata Schemas

“A significant need exists in
   many disciplines for long-                                                                   Message Bus
   term, distributed, and



                                                 Notifications
                                                                                                                          Workflow




                                      Workflow
                                                                                                                                   N            otification
                                                                                                                                                                 s
   stable data and metadata                                                                    Record
                                                                                                        Workflo
                                                                                                                w   Outputs
   repositories”
                                                                                                                                    Intermediate Results
                                                                                           Workflow Configuration and

   –   NSF Blue-Ribbon Advisory                                                                                   In   puts                                            Metadata Catalog
                                                                                                           rkflow
       Panel on Cyberinfrastructure                                                                 r d Wo
                                                                                               Reco
                                                                                                                                                         s
                                                                                                                                                   low




                                                                                                                                                                         ws




                                                                                                                                                                                     sults
                                                                                                                                            o   rkf




                                                                                                                                                                         lo
                                                                                                                                           W




                                                                                                                                                                       kf
                                       Workflow                                                                                       or




                                                                                                                                                                                          e
                                                                                                                                                                    or
                                                                                                                                    yF




                                                                                                                                                                                 Search R
                                                                                                                                                                  rW
                                                                 Co                                                         e   r
                                                                                                                         Qu




                                                                                                                                                                 ito
                                                                      mp




                                                                                                                                                             on
                                                                           os
“Metadata is key to being                                                    eW




                                                                                                                                                             M
                                                                               or
                                                                                    kfl
                                                                                          ow
  able to share results”
  –    UK e-Science Core Programme Study
                                                                                                                                                   Portal




                                 More Info: Scott Jensen
Applications
• LEAD
   – Lower entry barrier to using weather analysis tools
   – Improve detection, analysis & prediction of mesoscale weather
• Motif-Network
   – Transformation of sequenced genomes to “domain-space”
• Cyber-Infrastructure Evaluation
   – Performance evaluation of future supercomputer architectures
• ADAM
   – Algorithms for feature extraction, data normalization, classification
     and normalization
• GridChem
   – Molecular Chemistry Grid helping researchers run chemistry
     applications on Grid Environment

                        OGCE Workflow Tutorial
LEAD: A Weather Forecasting Workflow (1/2)
    Terrain data files
                                                                             NAM, RUC, GFS data                            9
                                         3                                                                                          3D Model Data
  1                                                                                                                                  Interpolator
          Terrain                                3D Model                                                                            (lateral Boundary
                                                    Data                       Surface data,                                             Conditions)
       Preprocessor                                                      upper air mesonet data and
                                                Interpolator
                                                (Initial Boundary            wind profiler data                  11                                  15
                                                   Conditions)
  2
                                                                                                                  ARPS to WRF                        IDV
         WRF Static                                                                                                    Data
        Preprocessor                                                                                               Interpolator
                                       4
                                                 88D Radar
                                                 Re-mapper
  Surface, terrestrial
                                                                                          7
       data files                                                                                                 10                    WRF
                                                                                    ADAS                                                 WRF
                                                                                                                    ARPS              12 WRFWRF
                                           Radar data
       Run once per                         (Level II)                                                             Ensemble
       forecast region                                              5                                              Generator
                                                                                                                                                         13
                                                NIDS Radar                            8
              Radar data                        Re-mapper                                                                      WRF to ARPS Data
               (Level III)                                                                ADAM                                   Interpolator

               Satellite                   6
                data                                                                                             Visualization on
                                                  Satellite Data                                                  users request
                                                   Re-mapper                                                                                             14
         Repeat                                                                                                                     ARPS Plotting
       periodically                                                                                                                   Program
       for new data                                                            Data mining:
                                                                              look for storm
                                                                                signature                                       Triggered if a storm
Static data     Real time data Initialization                                                  Forecast
                                                                                                                      13
                                                                                                          Visualization             is detected
                                                  Analysis     Data Mining
LEAD: A Weather Forecasting Workflow (2/2)




          WRF-Static running
            on Tungsten




                  OGCE Workflow Tutorial
Motif-Network: Whole Genome
               workflow
• Domain webs of large genomes
   – Input list of amino acid sequences
   – Identify all known domains
   – Construct webs
                                                Ensemble-type processing
                                                    (minimal network reqs)
                                                Capacity-type computing


                                                  Parallel processing
                                               Capability-type computing




                          Jeff Tilson, RENCI
CI: Execute Sub-Workflow

• Input a campaign step filename
• Execute GAMESS per step
  specification




                      Jeff Tilson, RENCI
Example: “Optimal” Weather
Prediction Using Dynamic Adaptivity
                                                 Storms Forming



                                                                      Forecast Model
Streaming
Observations                    Data Mining



          Instrument Steering

                                       Refine forecast grid



                                                                  On-Demand
                                                                  Grid Computing
Analyze &
              Predict



 Discover                    Research &
&Visualize                 Reproducibility




             Education &
              Outreach
Live Demo & Questions?




     OGCE Workflow Tutorial

More Related Content

What's hot

Mel McIntyre, OpenApp.ie LGMA
Mel McIntyre, OpenApp.ie LGMAMel McIntyre, OpenApp.ie LGMA
Mel McIntyre, OpenApp.ie LGMA
OpenSourceLGMA
 
クラウドいろは勉強会
クラウドいろは勉強会クラウドいろは勉強会
クラウドいろは勉強会
Daisuke Nakazawa
 
Session 24 - Distribute Data and Metadata Management with gLite
Session 24 - Distribute Data and Metadata Management with gLiteSession 24 - Distribute Data and Metadata Management with gLite
Session 24 - Distribute Data and Metadata Management with gLiteISSGC Summer School
 
Adapative Provisioning of Stream Processing Systems in the Cloud
Adapative Provisioning of Stream Processing Systems in the CloudAdapative Provisioning of Stream Processing Systems in the Cloud
Adapative Provisioning of Stream Processing Systems in the CloudJavier Cerviño
 
Cloud Biocep
Cloud BiocepCloud Biocep
Cloud Biocep
Inria
 
VectorBase DAS Tutorial
VectorBase DAS TutorialVectorBase DAS Tutorial
VectorBase DAS Tutorial
VectorBase
 
LMAX Architecture
LMAX ArchitectureLMAX Architecture
LMAX Architecture
Stephan Schmidt
 

What's hot (7)

Mel McIntyre, OpenApp.ie LGMA
Mel McIntyre, OpenApp.ie LGMAMel McIntyre, OpenApp.ie LGMA
Mel McIntyre, OpenApp.ie LGMA
 
クラウドいろは勉強会
クラウドいろは勉強会クラウドいろは勉強会
クラウドいろは勉強会
 
Session 24 - Distribute Data and Metadata Management with gLite
Session 24 - Distribute Data and Metadata Management with gLiteSession 24 - Distribute Data and Metadata Management with gLite
Session 24 - Distribute Data and Metadata Management with gLite
 
Adapative Provisioning of Stream Processing Systems in the Cloud
Adapative Provisioning of Stream Processing Systems in the CloudAdapative Provisioning of Stream Processing Systems in the Cloud
Adapative Provisioning of Stream Processing Systems in the Cloud
 
Cloud Biocep
Cloud BiocepCloud Biocep
Cloud Biocep
 
VectorBase DAS Tutorial
VectorBase DAS TutorialVectorBase DAS Tutorial
VectorBase DAS Tutorial
 
LMAX Architecture
LMAX ArchitectureLMAX Architecture
LMAX Architecture
 

Similar to Ogce Workflow Suite Tg09

libHPC: Software sustainability and reuse through metadata preservation
libHPC: Software sustainability and reuse through metadata preservationlibHPC: Software sustainability and reuse through metadata preservation
libHPC: Software sustainability and reuse through metadata preservationSoftwarePractice
 
Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical ISSGC Summer School
 
Session 49 Practical Semantic Sticky Note
Session 49 Practical Semantic Sticky NoteSession 49 Practical Semantic Sticky Note
Session 49 Practical Semantic Sticky NoteISSGC Summer School
 
Taverna workflows: provenance and reproducibility - STFC/NERC workshop 2013
Taverna workflows: provenance and reproducibility - STFC/NERC workshop 2013Taverna workflows: provenance and reproducibility - STFC/NERC workshop 2013
Taverna workflows: provenance and reproducibility - STFC/NERC workshop 2013
anpawlik
 
OpenStack Collaboration made in heaven with Heat, Mistral, Neutron and more..
OpenStack Collaboration made in heaven with Heat, Mistral, Neutron and more..OpenStack Collaboration made in heaven with Heat, Mistral, Neutron and more..
OpenStack Collaboration made in heaven with Heat, Mistral, Neutron and more..
Trinath Somanchi
 
Opal: Simple Web Services Wrappers for Scientific Applications
Opal: Simple Web Services Wrappers for Scientific ApplicationsOpal: Simple Web Services Wrappers for Scientific Applications
Opal: Simple Web Services Wrappers for Scientific Applications
Sriram Krishnan
 
Advances in Scientific Workflow Environments
Advances in Scientific Workflow EnvironmentsAdvances in Scientific Workflow Environments
Advances in Scientific Workflow Environments
Carole Goble
 
Data cleaning with the Kurator toolkit: Bridging the gap between conventional...
Data cleaning with the Kurator toolkit: Bridging the gap between conventional...Data cleaning with the Kurator toolkit: Bridging the gap between conventional...
Data cleaning with the Kurator toolkit: Bridging the gap between conventional...
Timothy McPhillips
 
Dell open stack powered cloud solution introduce & crowbar demo cosug-2012
Dell open stack powered cloud solution introduce & crowbar demo cosug-2012Dell open stack powered cloud solution introduce & crowbar demo cosug-2012
Dell open stack powered cloud solution introduce & crowbar demo cosug-2012OpenCity Community
 
Wolstencroft K - Workflows on the Cloud: scaling for national service
Wolstencroft K - Workflows on the Cloud: scaling for national serviceWolstencroft K - Workflows on the Cloud: scaling for national service
Wolstencroft K - Workflows on the Cloud: scaling for national service
Jan Aerts
 
Adam shiwa summerschool 2012
Adam shiwa summerschool 2012Adam shiwa summerschool 2012
Adam shiwa summerschool 2012aszbel
 
Team-Based Approach to Deploying VDI in Learning Environments
Team-Based Approach to Deploying VDI in Learning EnvironmentsTeam-Based Approach to Deploying VDI in Learning Environments
Team-Based Approach to Deploying VDI in Learning Environments
Jeremy Anderson
 
04.egovFrame Runtime Environment Workshop
04.egovFrame Runtime Environment Workshop04.egovFrame Runtime Environment Workshop
04.egovFrame Runtime Environment Workshop
Chuong Nguyen
 
The Very Very Latest in Database Development - Oracle Open World 2012
The Very Very Latest in Database Development - Oracle Open World 2012The Very Very Latest in Database Development - Oracle Open World 2012
The Very Very Latest in Database Development - Oracle Open World 2012
Lucas Jellema
 
The Very Very Latest In Database Development - Lucas Jellema - Oracle OpenWor...
The Very Very Latest In Database Development - Lucas Jellema - Oracle OpenWor...The Very Very Latest In Database Development - Lucas Jellema - Oracle OpenWor...
The Very Very Latest In Database Development - Lucas Jellema - Oracle OpenWor...
Getting value from IoT, Integration and Data Analytics
 
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Cloud-Native Apache Spark Scheduling with YuniKorn SchedulerCloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Databricks
 
vert.x - asynchronous event-driven web applications on the JVM
vert.x - asynchronous event-driven web applications on the JVMvert.x - asynchronous event-driven web applications on the JVM
vert.x - asynchronous event-driven web applications on the JVM
jbandi
 
Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?
Anton Nazaruk
 
Complex Er[jl]ang Processing with StreamBase
Complex Er[jl]ang Processing with StreamBaseComplex Er[jl]ang Processing with StreamBase
Complex Er[jl]ang Processing with StreamBase
darach
 

Similar to Ogce Workflow Suite Tg09 (20)

libHPC: Software sustainability and reuse through metadata preservation
libHPC: Software sustainability and reuse through metadata preservationlibHPC: Software sustainability and reuse through metadata preservation
libHPC: Software sustainability and reuse through metadata preservation
 
Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical
 
Session 49 Practical Semantic Sticky Note
Session 49 Practical Semantic Sticky NoteSession 49 Practical Semantic Sticky Note
Session 49 Practical Semantic Sticky Note
 
Taverna workflows: provenance and reproducibility - STFC/NERC workshop 2013
Taverna workflows: provenance and reproducibility - STFC/NERC workshop 2013Taverna workflows: provenance and reproducibility - STFC/NERC workshop 2013
Taverna workflows: provenance and reproducibility - STFC/NERC workshop 2013
 
OpenStack Collaboration made in heaven with Heat, Mistral, Neutron and more..
OpenStack Collaboration made in heaven with Heat, Mistral, Neutron and more..OpenStack Collaboration made in heaven with Heat, Mistral, Neutron and more..
OpenStack Collaboration made in heaven with Heat, Mistral, Neutron and more..
 
Opal: Simple Web Services Wrappers for Scientific Applications
Opal: Simple Web Services Wrappers for Scientific ApplicationsOpal: Simple Web Services Wrappers for Scientific Applications
Opal: Simple Web Services Wrappers for Scientific Applications
 
Advances in Scientific Workflow Environments
Advances in Scientific Workflow EnvironmentsAdvances in Scientific Workflow Environments
Advances in Scientific Workflow Environments
 
Data cleaning with the Kurator toolkit: Bridging the gap between conventional...
Data cleaning with the Kurator toolkit: Bridging the gap between conventional...Data cleaning with the Kurator toolkit: Bridging the gap between conventional...
Data cleaning with the Kurator toolkit: Bridging the gap between conventional...
 
System events concept presentation
System events concept presentationSystem events concept presentation
System events concept presentation
 
Dell open stack powered cloud solution introduce & crowbar demo cosug-2012
Dell open stack powered cloud solution introduce & crowbar demo cosug-2012Dell open stack powered cloud solution introduce & crowbar demo cosug-2012
Dell open stack powered cloud solution introduce & crowbar demo cosug-2012
 
Wolstencroft K - Workflows on the Cloud: scaling for national service
Wolstencroft K - Workflows on the Cloud: scaling for national serviceWolstencroft K - Workflows on the Cloud: scaling for national service
Wolstencroft K - Workflows on the Cloud: scaling for national service
 
Adam shiwa summerschool 2012
Adam shiwa summerschool 2012Adam shiwa summerschool 2012
Adam shiwa summerschool 2012
 
Team-Based Approach to Deploying VDI in Learning Environments
Team-Based Approach to Deploying VDI in Learning EnvironmentsTeam-Based Approach to Deploying VDI in Learning Environments
Team-Based Approach to Deploying VDI in Learning Environments
 
04.egovFrame Runtime Environment Workshop
04.egovFrame Runtime Environment Workshop04.egovFrame Runtime Environment Workshop
04.egovFrame Runtime Environment Workshop
 
The Very Very Latest in Database Development - Oracle Open World 2012
The Very Very Latest in Database Development - Oracle Open World 2012The Very Very Latest in Database Development - Oracle Open World 2012
The Very Very Latest in Database Development - Oracle Open World 2012
 
The Very Very Latest In Database Development - Lucas Jellema - Oracle OpenWor...
The Very Very Latest In Database Development - Lucas Jellema - Oracle OpenWor...The Very Very Latest In Database Development - Lucas Jellema - Oracle OpenWor...
The Very Very Latest In Database Development - Lucas Jellema - Oracle OpenWor...
 
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Cloud-Native Apache Spark Scheduling with YuniKorn SchedulerCloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
 
vert.x - asynchronous event-driven web applications on the JVM
vert.x - asynchronous event-driven web applications on the JVMvert.x - asynchronous event-driven web applications on the JVM
vert.x - asynchronous event-driven web applications on the JVM
 
Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?
 
Complex Er[jl]ang Processing with StreamBase
Complex Er[jl]ang Processing with StreamBaseComplex Er[jl]ang Processing with StreamBase
Complex Er[jl]ang Processing with StreamBase
 

More from smarru

Cyberinfrastructure Experiences with Apache Airavata
Cyberinfrastructure Experiences with Apache AiravataCyberinfrastructure Experiences with Apache Airavata
Cyberinfrastructure Experiences with Apache Airavata
smarru
 
Apache Airavata Credential Store
Apache Airavata Credential StoreApache Airavata Credential Store
Apache Airavata Credential Store
smarru
 
RESTLess Design with Apache Thrift: Experiences from Apache Airavata
RESTLess Design with Apache Thrift: Experiences from Apache AiravataRESTLess Design with Apache Thrift: Experiences from Apache Airavata
RESTLess Design with Apache Thrift: Experiences from Apache Airavata
smarru
 
Google Summer of Code at Apache Software Foundation
Google Summer of Code at Apache Software FoundationGoogle Summer of Code at Apache Software Foundation
Google Summer of Code at Apache Software Foundation
smarru
 
Gsoc airavata
Gsoc airavataGsoc airavata
Gsoc airavatasmarru
 
Learning Open Source through GSOC
Learning Open Source through GSOC Learning Open Source through GSOC
Learning Open Source through GSOC
smarru
 
Apache Student Induction ApacheCon 2013
Apache Student Induction ApacheCon 2013Apache Student Induction ApacheCon 2013
Apache Student Induction ApacheCon 2013
smarru
 
Apache Airavata ApacheCon2013
Apache Airavata ApacheCon2013Apache Airavata ApacheCon2013
Apache Airavata ApacheCon2013
smarru
 
Ogce Workflow Suite
Ogce Workflow SuiteOgce Workflow Suite
Ogce Workflow Suitesmarru
 

More from smarru (9)

Cyberinfrastructure Experiences with Apache Airavata
Cyberinfrastructure Experiences with Apache AiravataCyberinfrastructure Experiences with Apache Airavata
Cyberinfrastructure Experiences with Apache Airavata
 
Apache Airavata Credential Store
Apache Airavata Credential StoreApache Airavata Credential Store
Apache Airavata Credential Store
 
RESTLess Design with Apache Thrift: Experiences from Apache Airavata
RESTLess Design with Apache Thrift: Experiences from Apache AiravataRESTLess Design with Apache Thrift: Experiences from Apache Airavata
RESTLess Design with Apache Thrift: Experiences from Apache Airavata
 
Google Summer of Code at Apache Software Foundation
Google Summer of Code at Apache Software FoundationGoogle Summer of Code at Apache Software Foundation
Google Summer of Code at Apache Software Foundation
 
Gsoc airavata
Gsoc airavataGsoc airavata
Gsoc airavata
 
Learning Open Source through GSOC
Learning Open Source through GSOC Learning Open Source through GSOC
Learning Open Source through GSOC
 
Apache Student Induction ApacheCon 2013
Apache Student Induction ApacheCon 2013Apache Student Induction ApacheCon 2013
Apache Student Induction ApacheCon 2013
 
Apache Airavata ApacheCon2013
Apache Airavata ApacheCon2013Apache Airavata ApacheCon2013
Apache Airavata ApacheCon2013
 
Ogce Workflow Suite
Ogce Workflow SuiteOgce Workflow Suite
Ogce Workflow Suite
 

Recently uploaded

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 

Recently uploaded (20)

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 

Ogce Workflow Suite Tg09

  • 1. OGCE WorkflowSuite for Science Gateways Suresh Marru, Raminder Singh, Chathura Herath & Marlon Pierce Indiana University
  • 2. OGCE Gateways TeraGrid User Portal (LEAD, GridChem, …) TG GIG Generalize, Harden, Build Test Gateways/E-Science Community
  • 3. Requirements from gateways • Gateways demand scientific workflow systems to be: – Flexible – Dynamic – Interactive – Technology Adaptive – Interoperable with Emerging Computational Resources and their job management interfaces
  • 4. OGCE Workflow Suite • Generic Service Toolkit – Tool to wrap command-line applications as web services – Handles file staging&job submissions – Extensible runtime for security, resource brokering& urgent computing – Generic Factory service for on-demand creation of application services • XRegistry – Information repository for the OGCE workflow suite – Register, search, retrieve&share XML documents – User & hierarchical group based authorization • XBaya – GUI based tool to compose&monitor workflows – Extensible support for compiler plug-ins like BPEL &Jython – Dynamic Workflow Execution support to start, pause, resume, rewind of workflow executions OGCE Workflow Tutorial
  • 5. Features • Security – Authentication and authorization – Secure invocations between services – Support for gateway community accounts – Support for multiple user accounts • Reliability – Retry job submissions and file staging – Fault Tolerance and Recovery service • Over-provisioning and migration • Compatibility – Taverna, Kepler and Trianna OGCE Workflow Tutorial
  • 6. Application Services • Workflows are built by composing web Application Factory services c – Fortran applications are “wrapped” by a Application Factory which generates a web service for the app. • Registers WSDL for the service with a registry App – Each service generates a stream of Service notifications that log the service actions back to the XMC Cat Metadata Catalog. Run program & publish events
  • 7. Workflow Composition, Execution & Monitoring Baya enables users to construct, share, execute and monitor sequence of tasks executing on their local workstations to high-end compute resources.
  • 8. Service Monitoring via Events • The service output is a stream of events Application Service – I am running your request Instance – I have started to move your input files. – I have all the files 6 5 – I am running your application. 4 – The application is finished 3 – I am moving the output to you file space 2 1 – I am done. • These are automatically generated by the service using a distributed event system (WS-Eventing / WS-Notification) Notification – Topic based pub-sub system with Channel a well known “channel”. Subscribe Topic=x x x listener publisher
  • 9. OGCE Workflow Tools WRF-Static running on Tungsten
  • 11. XML Metadata Catalog (XMC Cat) Taming Complex Scientific Metadata Schemas “A significant need exists in many disciplines for long- Message Bus term, distributed, and Notifications Workflow Workflow N otification s stable data and metadata Record Workflo w Outputs repositories” Intermediate Results Workflow Configuration and – NSF Blue-Ribbon Advisory In puts Metadata Catalog rkflow Panel on Cyberinfrastructure r d Wo Reco s low ws sults o rkf lo W kf Workflow or e or yF Search R rW Co e r Qu ito mp on os “Metadata is key to being eW M or kfl ow able to share results” – UK e-Science Core Programme Study Portal More Info: Scott Jensen
  • 12. Applications • LEAD – Lower entry barrier to using weather analysis tools – Improve detection, analysis & prediction of mesoscale weather • Motif-Network – Transformation of sequenced genomes to “domain-space” • Cyber-Infrastructure Evaluation – Performance evaluation of future supercomputer architectures • ADAM – Algorithms for feature extraction, data normalization, classification and normalization • GridChem – Molecular Chemistry Grid helping researchers run chemistry applications on Grid Environment OGCE Workflow Tutorial
  • 13. LEAD: A Weather Forecasting Workflow (1/2) Terrain data files NAM, RUC, GFS data 9 3 3D Model Data 1 Interpolator Terrain 3D Model (lateral Boundary Data Surface data, Conditions) Preprocessor upper air mesonet data and Interpolator (Initial Boundary wind profiler data 11 15 Conditions) 2 ARPS to WRF IDV WRF Static Data Preprocessor Interpolator 4 88D Radar Re-mapper Surface, terrestrial 7 data files 10 WRF ADAS WRF ARPS 12 WRFWRF Radar data Run once per (Level II) Ensemble forecast region 5 Generator 13 NIDS Radar 8 Radar data Re-mapper WRF to ARPS Data (Level III) ADAM Interpolator Satellite 6 data Visualization on Satellite Data users request Re-mapper 14 Repeat ARPS Plotting periodically Program for new data Data mining: look for storm signature Triggered if a storm Static data Real time data Initialization Forecast 13 Visualization is detected Analysis Data Mining
  • 14. LEAD: A Weather Forecasting Workflow (2/2) WRF-Static running on Tungsten OGCE Workflow Tutorial
  • 15. Motif-Network: Whole Genome workflow • Domain webs of large genomes – Input list of amino acid sequences – Identify all known domains – Construct webs Ensemble-type processing (minimal network reqs) Capacity-type computing Parallel processing Capability-type computing Jeff Tilson, RENCI
  • 16. CI: Execute Sub-Workflow • Input a campaign step filename • Execute GAMESS per step specification Jeff Tilson, RENCI
  • 17. Example: “Optimal” Weather Prediction Using Dynamic Adaptivity Storms Forming Forecast Model Streaming Observations Data Mining Instrument Steering Refine forecast grid On-Demand Grid Computing
  • 18. Analyze & Predict Discover Research & &Visualize Reproducibility Education & Outreach
  • 19. Live Demo & Questions? OGCE Workflow Tutorial