This document summarizes a presentation on the WS-VLAM Workflow Management System and its applications. The presentation introduces WS-VLAM, describing its features for composing and executing hierarchical workflows across distributed computing resources. It also compares WS-VLAM to other workflow systems and discusses WS-VLAM's support for task farming, auto-scaling of workflows, exposing workflows as services, and meeting deadlines through on-demand resource provisioning from multiple cloud providers.
Natural Gas Supply and Demand: A Delicate Balancing ActPointLogicEnergy
Jack Weixel, PointLogic Energy's vice president of analysis, delivered this presentation, "Natural Gas Supply and Demand: A Delicate Balancing Act," to the attendees of the Mid-Continent LDC Gas Forum on September 14, 2015.
Natural Gas Supply and Demand: A Delicate Balancing ActPointLogicEnergy
Jack Weixel, PointLogic Energy's vice president of analysis, delivered this presentation, "Natural Gas Supply and Demand: A Delicate Balancing Act," to the attendees of the Mid-Continent LDC Gas Forum on September 14, 2015.
Simulation of Heterogeneous Cloud InfrastructuresCloudLightning
During the last years, except from the traditional CPU based hardware servers, hardware accelerators are widely used in various HPC application areas. More specifically, Graphics Processing Units (GPUs), Many Integrated Cores (MICs) and Field-Programmable Gate Arrays (FPGAs) have shown a great potential in HPC and have been widely mobilised in supercomputing and in HPC-Clouds. This presentation focuses on the development of a cloud simulation framework that supports hardware accelerators. The design and implementation of the framework are also discussed.
This presentation was given by Dr. Konstantinos Giannoutakis (CERTH) at the CloudLightning Conference on 11th April 2017.
Opal: Simple Web Services Wrappers for Scientific ApplicationsSriram Krishnan
The grid-based infrastructure enables large-scale scientific applications to be run on distributed resources and coupled in innovative ways. However, in practice, grid resources are not very easy to use for the end-users who have to learn how to generate security credentials, stage inputs and outputs, access grid-based schedulers, and install complex client software. There is an imminent need to provide transparent access to these resources so that the end-users are shielded from the complicated details, and free to concentrate on their domain science. Scientific applications wrapped as Web services alleviate some of these problems by hiding the complexities of the back-end security and computational infrastructure, only exposing a simple SOAP API that can be accessed programmatically by application-specific user interfaces. However, writing the application services that access grid resources can be quite complicated, especially if it has to be replicated for every application. In this presentation, we present Opal which is a toolkit for wrapping scientific applications as Web services in a matter of hours, providing features such as scheduling, standards-based grid security and data management in an easy-to-use and configurable manner
Dr. Konstantinos Giannoutakis presents the CloudLightning simulator, a bespoke cloud simulation engine built for modelling and simulating heterogeneous resources as well as self-organising systems.
This presentation was given at the CloudLightning Conference held in conjunction with NC4 2017 in Dublin City University on 11th April 2017.
Creating a Centralized Consumer Profile Management Service with WebSphere Dat...Prolifics
In this presentation will talk about how one of the world's leading Financial Institutions, leveraged WebSphere DataPower to provide a set of centralized consumer profile management services. This central service would be leveraged by internal and external applications, and would align with enterprise marketing capabilities. The solution included a complex security model which included the following products: Tivoli Directory Server, Tivoli Access Manager and Tivoli Federated Identity Manager. We will describe how to build complex orchestrations in WebSphere DataPower, and also go through some of the performance tuning options we implemented to achieve a high degree of efficiency.
DOTNET 2013 IEEE CLOUDCOMPUTING PROJECT Dynamic resource allocation using vir...IEEEGLOBALSOFTTECHNOLOGIES
To Get any Project for CSE, IT ECE, EEE Contact Me @ 09849539085, 09966235788 or mail us - ieeefinalsemprojects@gmail.com-Visit Our Website: www.finalyearprojects.org
Software Architecture for Cloud InfrastructureTapio Rautonen
Distributed systems are hard to build. Software architecture must be carefully crafted to suit cloud infrastructure.
Design for failure. Learn from failure. Adopt new cloud compatible design patterns and follow the guidelines during the journey of building cloud native applications.
Highly available and scalable web hosting can be complex and expensive. Learn how Amazon Web Services provides the reliable, scalable, secure, and high performance infrastructure required for web applications while enabling an elastic, scale out and scale down infrastructure to match IT costs in real time as customer traffic fluctuates.
How jKool Analyzes Streaming Data in Real Time with DataStaxDataStax
In this webinar, Charles Rich, VP of Product Management at jKool will share their journey with DataStax; how jKool knew from the start that traditional relational databases wouldn’t work for the scalability and availability demands of time-series data, and why they turned to DataStax Enterprise for blazing performance and powerful enterprise search and analytics capabilities.
How jKool Analyzes Streaming Data in Real Time with DataStaxjKool
jKool provides an application analytics SaaS for DevOps. These slides illustrate some of the choices we had to make and the architectural decisions to build a system for both real-time and historical application analytics.
I presented "Cloudsim & Green Cloud" in First National Workshop of Cloud Computing at Amirkabir University on 31st October and 1st November, 2012.
Enjoy it!
OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs
AI has been a hot topic lately, with advances being made constantly in what is possible, there has not been as much discussion of the infrastructure and scaling challenges that come with it. How do you support dozens of different languages and frameworks, and make them interoperate invisibly? How do you scale to run abstract code from thousands of different developers, simultaneously and elastically, while maintaining less than 15ms of overhead?
At Algorithmia, we’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework (from scikit-learn to tensorflow). We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them.
In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI” – a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.
This chapter discusses various classification attributed to parallel architectures. It also introduces related parallel programming models and presents the actions of these models on parallel architectures. Notions such as Data parallelism Task parallelism, Tighty and Coupled system, UMA/NUMA, Multicore computing, Symmetric multiprocessing, Distributed Computing, Cluster computing, Shared memory without thread/Thread, etc..
Precima data scientist and architect discussing their data science and big data tech stack at the Toronto Data Science & Big Data meetup on Jan 30, 2019 hosted by WeCloudData https://weclouddata.com and sponsored by precima and loyaltyone.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Simulation of Heterogeneous Cloud InfrastructuresCloudLightning
During the last years, except from the traditional CPU based hardware servers, hardware accelerators are widely used in various HPC application areas. More specifically, Graphics Processing Units (GPUs), Many Integrated Cores (MICs) and Field-Programmable Gate Arrays (FPGAs) have shown a great potential in HPC and have been widely mobilised in supercomputing and in HPC-Clouds. This presentation focuses on the development of a cloud simulation framework that supports hardware accelerators. The design and implementation of the framework are also discussed.
This presentation was given by Dr. Konstantinos Giannoutakis (CERTH) at the CloudLightning Conference on 11th April 2017.
Opal: Simple Web Services Wrappers for Scientific ApplicationsSriram Krishnan
The grid-based infrastructure enables large-scale scientific applications to be run on distributed resources and coupled in innovative ways. However, in practice, grid resources are not very easy to use for the end-users who have to learn how to generate security credentials, stage inputs and outputs, access grid-based schedulers, and install complex client software. There is an imminent need to provide transparent access to these resources so that the end-users are shielded from the complicated details, and free to concentrate on their domain science. Scientific applications wrapped as Web services alleviate some of these problems by hiding the complexities of the back-end security and computational infrastructure, only exposing a simple SOAP API that can be accessed programmatically by application-specific user interfaces. However, writing the application services that access grid resources can be quite complicated, especially if it has to be replicated for every application. In this presentation, we present Opal which is a toolkit for wrapping scientific applications as Web services in a matter of hours, providing features such as scheduling, standards-based grid security and data management in an easy-to-use and configurable manner
Dr. Konstantinos Giannoutakis presents the CloudLightning simulator, a bespoke cloud simulation engine built for modelling and simulating heterogeneous resources as well as self-organising systems.
This presentation was given at the CloudLightning Conference held in conjunction with NC4 2017 in Dublin City University on 11th April 2017.
Creating a Centralized Consumer Profile Management Service with WebSphere Dat...Prolifics
In this presentation will talk about how one of the world's leading Financial Institutions, leveraged WebSphere DataPower to provide a set of centralized consumer profile management services. This central service would be leveraged by internal and external applications, and would align with enterprise marketing capabilities. The solution included a complex security model which included the following products: Tivoli Directory Server, Tivoli Access Manager and Tivoli Federated Identity Manager. We will describe how to build complex orchestrations in WebSphere DataPower, and also go through some of the performance tuning options we implemented to achieve a high degree of efficiency.
DOTNET 2013 IEEE CLOUDCOMPUTING PROJECT Dynamic resource allocation using vir...IEEEGLOBALSOFTTECHNOLOGIES
To Get any Project for CSE, IT ECE, EEE Contact Me @ 09849539085, 09966235788 or mail us - ieeefinalsemprojects@gmail.com-Visit Our Website: www.finalyearprojects.org
Software Architecture for Cloud InfrastructureTapio Rautonen
Distributed systems are hard to build. Software architecture must be carefully crafted to suit cloud infrastructure.
Design for failure. Learn from failure. Adopt new cloud compatible design patterns and follow the guidelines during the journey of building cloud native applications.
Highly available and scalable web hosting can be complex and expensive. Learn how Amazon Web Services provides the reliable, scalable, secure, and high performance infrastructure required for web applications while enabling an elastic, scale out and scale down infrastructure to match IT costs in real time as customer traffic fluctuates.
How jKool Analyzes Streaming Data in Real Time with DataStaxDataStax
In this webinar, Charles Rich, VP of Product Management at jKool will share their journey with DataStax; how jKool knew from the start that traditional relational databases wouldn’t work for the scalability and availability demands of time-series data, and why they turned to DataStax Enterprise for blazing performance and powerful enterprise search and analytics capabilities.
How jKool Analyzes Streaming Data in Real Time with DataStaxjKool
jKool provides an application analytics SaaS for DevOps. These slides illustrate some of the choices we had to make and the architectural decisions to build a system for both real-time and historical application analytics.
I presented "Cloudsim & Green Cloud" in First National Workshop of Cloud Computing at Amirkabir University on 31st October and 1st November, 2012.
Enjoy it!
OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs
AI has been a hot topic lately, with advances being made constantly in what is possible, there has not been as much discussion of the infrastructure and scaling challenges that come with it. How do you support dozens of different languages and frameworks, and make them interoperate invisibly? How do you scale to run abstract code from thousands of different developers, simultaneously and elastically, while maintaining less than 15ms of overhead?
At Algorithmia, we’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework (from scikit-learn to tensorflow). We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them.
In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI” – a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.
This chapter discusses various classification attributed to parallel architectures. It also introduces related parallel programming models and presents the actions of these models on parallel architectures. Notions such as Data parallelism Task parallelism, Tighty and Coupled system, UMA/NUMA, Multicore computing, Symmetric multiprocessing, Distributed Computing, Cluster computing, Shared memory without thread/Thread, etc..
Precima data scientist and architect discussing their data science and big data tech stack at the Toronto Data Science & Big Data meetup on Jan 30, 2019 hosted by WeCloudData https://weclouddata.com and sponsored by precima and loyaltyone.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Securing your Kubernetes cluster_ a step-by-step guide to success !
Adam e science_2013_1
1. • Clck
to
edit
Master
0tle
style
• Click
to
edit
Master
0tle
style
WS-‐VLAM
Workflow
Management
System
and
its
Applica0ons
Adam
Belloum
Ins0tute
of
Informa0cs
University
of
Amsterdam
a.s.z.belloum@uva.nl
Lunch meeting, Netherlands eScience Center, Amsterdam 2013
2. • Clck
to
edit
Master
0tle
style
Outline
• Introduc0on
–
Life
cycle
of
e-‐Science
Workflow
• Different
approaches
to
workflow
scheduling
– Workflow
Process
Modeling
&
Management
In
Grid/
Cloud
– Workflow
and
Web
services
(intrusive/non
intrusive)
• Provenance
• Compu0ng
in
the
browser
• Conclusions
3. • Clck
to
edit
Master
0tle
style
Workflow
Management
Systems
Workflow
management
system
coordinates
the
execu/on
of
a
scien0fic
applica0ons
on
a
set
of
compu/ng
distributed
resources
Compu0ng
task
management
Grid/Cloud
middleware
Data
Security
Data
provenance
Workflow
Engine
Repositories
Data Management
http://www.youtube.com/watch v=R6bTFrzaR_w&feature=player_embedded
4. • Clck
to
edit
Master
0tle
style
Life
Cycle
of
a
Scien0fic
Experiment
(1) Problem
investigation:
• Look for relevant problems
• Browse available tools
• Define the goal
• Decompose into steps
(2) Experiment
Prototyping:
• Design experiment workflows
• Develop necessary components
(3) Experiment
Execution:
• Execute experiment processes
• Control the execution
• Collect and analysis data
(4) Results
Publication:
• Annotate data
• Publish data
Shared
repositories
Collaborative e-Science experiments: from scientific workflow to knowledge sharing A.S.Z. Belloum, Vladimir
Korkhov, Spiros Koulouzis, Marcia A Inda, and Marian Bubak JULY/AUGUSTInternet Computing, IEEE, vol.15, no.4, pp.39-47,
July-Aug. 2011 doi: 10.1109/MIC.2011.87
5. • Clck
to
edit
Master
0tle
style
Model
of
computa0on
• Target:
stream-‐based
Applica0ons
– Engine
co-‐allocates
all
workflow
components
• Communica0on:
0me
coupled
– Assumes
components
are
running
– Simultaneously
– Synchronized
p2p
WS-‐VLAM
Engine:
Architecture
(1st
genera0on)
Service host(s) compute element(s)
GRAM
services
GT4 Java Container
RTSM Factory
Delegation service
Worker nodes
Pre/ws-GRAM/GT5
Client
Delegate
RTSM
Instance
Workflow
components
6. • Clck
to
edit
Master
0tle
style
WS-‐VLAM
Features
• Provide
streaming
facili0es
between
applica0ons
executed
on
resource
geographically
distributed.
• Composi/on
and
the
execu0on
of
hierarchical
workflows.
• Remote
graphical
output.
• Detach/a:ach
capability
for
long
running
workflows.
• Provides
a
monitoring
facili0es
based
on
the
WS-‐no0fica0on.
• Provides
workflow
farming
possibili0es.
More features: http://staff.science.uva.nl/~gvlam/wsvlam/demos/wsvlam-about.html
SigWin-Detector workflow has been developed in the VL-e project to detect ridges in
for instance a Gene Expression sequence or Human transcriptome map, BMC Research
Notes 2008, 1:63 doi:10.1186/1756-0500-1-63.
DNA curvature of the Escherichia Coli chromosome
7. • Clck
to
edit
Master
0tle
style
Easy
Deployment
czxc`
Workflow composer (java Web Start)
• WS-VLAM composer
• VBrowser
• Semantic tools
SAW: Semantic Annotation for Workflow
CLAMP: Connecting LAnguage for Modules & Programs
HAMMER: Hybrid-bAsed MatchMaker for e-Science
Resources
Sara: National super
computing center
Server host
Production Grid
Experimental
Environment
Storage
WSRF Services (2 services)
- WS-VLAM engine
- workflow component repository
8. • Clck
to
edit
Master
0tle
style
WS-VLAM composer
• Workflow engine may be
invoked form other
systems like
§ Taverna, Kepler, Pgrade
• Workflow may be made
available to entire
community
§ using Web 2.0 approach
Workflow
Sharing
and
re-‐usability
9. • Clck
to
edit
Master
0tle
style
Comparing
to
other
Workflow
Management
Systems
• WS-‐VLAM
was
studied
by
Elts
and
Bungartz
in
2010
from
the
Ins0tute
of
informa0cs
of
the
Technical
university
of
Munich
as
a
poten0al
pla]orm
for
a
PhD
work
Grid-Workflow-Management-Systeme fur die
Ausfuhrung wissenschaftlicher Prozessablaufe
Ekaterina Elts, Hans-Joachim Bungartz
http://staff.science.uva.nl/~gvlam/wsvlam/
Publications/workflow_review_Elts.pdf
10. • Clck
to
edit
Master
0tle
style
WS-‐VLAM
Comparing
to
other
Workflow
Management
Systems
by Ekaterina Elts TUMunchen
11. • Clck
to
edit
Master
0tle
style
WS-‐VLAM
Comparing
to
other
Workflow
Management
Systems
by Ekaterina Elts TUMunchen
12. • Clck
to
edit
Master
0tle
style
WS-‐VLAM
Engine:
architecture
(2nd
genera0on)
• Target:
loosely
couple
applica0ons
– components
scheduled
depending
on
data
– components
only
ac/vated
when
data
is
available
– no
need
for
co-‐alloca/on
• Communica0on:
0me
decouples
– messaging
communica0on
system.
– components
not
synchronized
13. • Clck
to
edit
Master
0tle
style
WS-‐VLAM
Engine:
Architecture
(2nd
genera0on)
Data
driven
Workflow
coordina0on
Reginald Cushing, Spiros Koulouzis, Adam S. Z. Belloum, Marian Bubak, Prediction-based Auto-scaling
of Scientific Workflows, MGC’2011, December 12th, 2011, Lisbon, Portugal
14. • Clck
to
edit
Master
0tle
style
Farming
with
WS-‐VLAM
• Task
farming:
task
replica0on
– parameter
sweep
applica0on,
– DNA
Sequencing,
– Monte
Carlo,
– …
•
Implements
3
types
of
farming:
– Auto
Farming:
the
number
of
tasks/services
to
run
is
propor0onal
to
the
load
– One-‐to-‐One
Farming:
A
task
replicated
for
every
message
received.
– Fixed
Farming:
user
defined
farming.
15. • Clck
to
edit
Master
0tle
style
Farming
and
Auto-‐scaling
of
Workflows
Reginald Cushing, Spiros Koulouzis, Adam S. Z. Belloum, Marian Bubak, Prediction-based Auto-scaling
of Scientific Workflows, Proceedings of the 9th International Workshop on Middleware for Grids, Clouds
and e-Science, ACM/IFIP/USENIX December 12th, 2011, Lisbon, Portugal
16. • Clck
to
edit
Master
0tle
style
Workflow
as
a
Service
(WFaaS)
Reduce
Scheduling
Overhead
Workflow as a Service: An Approach to Workflow Farming, Reginald Cushing, Adam S. Z.
Belloum, V. Korkhov, D. Vasyunin, M.T. Bubak, C. Leguy ECMLS’12, June 18, 2012, Delft,
The Netherlands
17. • Clck
to
edit
Master
0tle
style
Resource
on-‐demand
for
Applica0ons
with
Hard
Deadline
(Urgent
Compu0ng)
Resource on-demand using multiple cloud providers, Super-computing 2010, and SCALE 2012
18. • Clck
to
edit
Master
0tle
style
Outline
(1)
Problem
investigation:
(2)
Experiment
Prototyping
(3)
Experiment
Execution:
(4)
Results
Publication:
Shared
repositories
• Introduc0on
–
Life
cycle
of
e-‐Science
Workflow
• Different
approaches
to
workflow
scheduling
– Workflow
Process
Modeling
&
Management
In
Grid/Cloud
– Workflow
and
Web
services
(intrusive/non
intrusive)
• Provenance
• Compu0ng
in
the
browser
• Conclusions
19. • Clck
to
edit
Master
0tle
style
Web
Services
in
eScience
with
WS-‐VLAM
• WS
offer
interoperability
and
flexibility
in
a
large
scale
distributed
environment.
• WS
can
be
combined
in
a
workflow
so
that
more
complex
opera0ons
may
be
achieved,
but
any
workflow
implementa0on
is
poten0ally
faced
with
a
data
transport
problem
or
being
overload
Scale
up
the
number
of
Web
service
to
keep
up
with
the
incoming
load
20. • Clck
to
edit
Master
0tle
style
Scaling
up
the
number
of
Web
services
• Tasks/Jobs
can
be
queued
on
the
runqueue.
• The
service
submission
listens
on
the
runqueue
and
picks
up
new
tasks
to
submit
• Resources
such
as
Grid
or
Cloud
are
abstracted
using
submieers
plugins
– Enabling
a
new
resource
is
a
maeer
of
wri0ng
its
submieer
(Condor,
ibis,
…)
Reginald Cushing, Spiros Koulouzis, Adam S. Z. Belloum, Marian Bubak, Dynamic Handling for
Cooperating Scientific Web Services, 7th IEEE International Conference on e-Science, December 2011,
Stockholm, Sweden
21. • Clck
to
edit
Master
0tle
style
Sequence
Alignment
Use
Case
• Workflow
with
2
pipelines.
The
pipelines
perform
sequence
alignments
using
data
from
UniProtKB
• Each
pipeline
performs
22500
alignments
i.e.
45100
total
alignments
in
all
• All
modules
are
standard
web
services
which
are
hosted
in
the
modified
Axis2
container
• The
alignments
where
performed
using
BioJava
api
• Source
and
sink
are
part
of
the
bootstrapping
sequence.
– Source
submits
the
getSequenceId
service
– while
sink
waits
for
output
from
the
htmlRenderer
Reginald Cushing, Spiros Koulouzis, Adam S. Z. Belloum, Marian Bubak, Dynamic Handling for
Cooperating Scientific Web Services, 7th IEEE International Conference on e-Science, December 2011,
Stockholm, Sweden
22. • Clck
to
edit
Master
0tle
style
Scale
up
web
services
Peaks
in
load(lej)
will
result
in
peaks
in
instances(right).
The
fuzzy
controllers
scale
up
the
web
services
to
meet
the
demands
23. • Clck
to
edit
Master
0tle
style
Enabling
web
services
to
consume
and
produce
large
distributed
• In
service
orchestra0on,
all
data
is
passed
to
the
workflow
engine
before
delivered
to
a
consuming
WS
• Data
transfers
are
made
through
SOAP,
which
is
unfit
for
large
data
transfers
Enabling web services to consume and produce large distributed datasets Spiros
Koulouzis, Reginald Cushing, Konstantinos Karasavvas, Adam Belloum, Marian Bubak to be
published JAN/FEB, IEEE Internet Computing, 2012
24. • Clck
to
edit
Master
0tle
style
Enabling
web
services
to
consume
and
produce
large
distributed
Enabling web services to consume and produce large distributed datasets Spiros
Koulouzis, Reginald Cushing, Konstantinos Karasavvas, Adam Belloum, Marian Bubak to be
published JAN/FEB, IEEE Internet Computing, 2012
Indexing Web Services for Information
Retrieval (NER) are tools that help
biologists to identify and retrieve information
• Index 8.4GB of medline documents
25. • Clck
to
edit
Master
0tle
style
Outline
(1)
Problem
investigation:
(2)
Experiment
Prototyping
(3)
Experiment
Execution:
(4)
Results
Publication:
Shared
repositories
• Introduc0on
–
Life
cycle
of
e-‐Science
Workflow
• Different
approaches
to
workflow
scheduling
– Workflow
Process
Modeling
&
Management
In
Grid/Cloud
– Workflow
and
Web
services
(intrusive/non
intrusive)
• Provenance
• Compu0ng
in
the
browser
• Conclusions
26. • Clck
to
edit
Master
0tle
style
Provenance/
Reproducibility
• “A
complete
provenance
record
for
a
data
object
allows
the
possibility
to
reproduce
the
result
and
reproducibility
is
a
cri0cal
component
of
the
scien0fic
method”
• Provenance:
The
recording
of
metadata
and
provenance
informa0on
during
the
various
stages
of
the
workflow
lifecycle
Workflows and e-Science: An overview of workflow system features
and capabilities Ewa Deelmana, Dennis Gannonb, Matthew Shields c, Ian Taylor, Future
Generation Computer Systems 25 (2009) 528540
27. • Clck
to
edit
Master
0tle
style
History-‐tracing
XML
(FH
Aachen)
provides
data/process
provenance
following
an
approach
that
• maps
the
workflow
graph
to
a
layered
structure
of
an
XML
document.
• This
allows
an
intui0ve
and
easy
processable
representa0on
of
the
workflow
execu0on
path
• Workflow
components
can
be
eventually,
electronically
signed.
<patternMatch>
<events>
<PortResolved>provenance data</PortResolved>
<ConDone> provenance data </ConDone>
...
</events>
<fileReader2>
<events> ... </events>
<sign-fileReader2> ... </sign-fileReader2>
</fileReader2>
<sffToFasta>
Reference
</sffToFasta>
<sign-patternMatch> ... </sign-patternMatch>
</patternMatch>
M. Gerards, Adam S. Z. Belloum, F. Berritz, V. Snder, S. Skorupa, A History-tracing XML-base
Provenance Framework for workflows, WORKS 2010, New Orleans, USA, November 2010
28. • Clck
to
edit
Master
0tle
style
[Biomedical engineering Cardiovascular
biomechanics group TUE])
wave propagation model of blood flow in large
vessels using an approximate velocity profile
function:
a biomedical study for which 3000 runs were
required to perform a global sensitivity analysis
of a blood pressure wave propagation in arteries
User Interface to compose workflow (top
right), monitor the execution of the farmed
workflows (top left), and monitor each run
separately (bottom left) data
Query interface for the provenance data
collected from 3000 simulations of the
“wave propagation model of blood flow in
large vessels using an approximate velocity
profile function”
BigGrid project 2009, presented EGI/
BigGrid technical forum 2010
Wave
Propaga0on
in
Blood
Flow
29. • Clck
to
edit
Master
0tle
style
Alignment
of
DNA
Sequence
(Blast)
For Each workflow run
• The provenance data is collected an stored
following the XML-tracing system
• User interface allows to reproduce events that
occurred at runtime (replay mode)
• User Interface can be customized (User can
select the events to track)
• User Interface show resource usage
The aim of the application is the alignment
of DNA sequence data with a given
reference database.
on-going work UvA-AMC-fh-aachen
[Department of Clinical
Epidemiology, Biostatistics and
Bioinformatics (KEBB), AMC ]
30. • Clck
to
edit
Master
0tle
style
Sensi0vity
Analysis
of
Cardiovascular
Models
Study covers: 164 patient-specific input parameters
(i.e. vessel diameter and length),
• 1494000 Monte Carlo runs are needed for this
first analysis.
• Expected execution time on a PC 14.525 hours
(20 Months)
FP7 Project VPH-Share " Virtual Physiological
Human: Sharing for Healthcare”
32. • Clck
to
edit
Master
0tle
style
Outline
(1)
Problem
investigation:
(2)
Experiment
Prototyping
(3)
Experiment
Execution:
(4)
Results
Publication:
Shared
repositories
• Introduc0on
–
Life
cycle
of
e-‐Science
Workflow
• Different
approaches
to
workflow
scheduling
– Workflow
Process
Modeling
&
Management
In
Grid/Cloud
– Workflow
and
Web
services
(intrusive/non
intrusive)
• Provenance
• Compu0ng
in
the
browser
• Conclusions
33. • Clck
to
edit
Master
0tle
style
Compu0ng
in
the
Browser
(different
approach
to
Grid-‐desktop)
• Objec/ves
– Distributed
compu0ng
using
web
browsers
• Features:
– volunteer
compu0ng
instantly
(no
third
party
sojware
installa0on)
• How
does
it
work:
– Social
media
mediates
the
trust
between
the
user
and
the
volunteers
asked
to
join
the
network.
• A
user
with
a
distributed
applica0on
uses
social
media
to
get
colleagues
and
friends
to
donate
CPU
• Colleagues
and
friends
join
the
network
by
simply
opening
the
shared
URL.
•
Compu/ng
starts
almost
instantly.
34. • Clck
to
edit
Master
0tle
style
Compu0ng
in
the
Browser
Distributed computing on an Ensemble of Browsers, R. Cushing, G.a Putra, S. Koulouzis, A.S.Z Belloum,
M.T. Bubak, C. de Laat IEEE Internet Computing, PrePress 10.1109/MIC.2013.3, January 2013
Application
• Computing 33,000 bio-informatics tasks on
the global cluster of browsers
• Announcing the experiment using social
media: via social media tools: twitter,
FaceBook, Linkedin, and project mailing lists.
• Volunteers were asked to open the Weevil web
page http://elab.lab.uvalight.net/~weevil/
and agree to donate their CPU for 3 hours on
Friday December 2011 from 12:00-14:00
0.4
0.6
0.8
1
1.2
1.4
00:00 01:00 02:00 03:00 04:00 05:00
0.0e+00
5.0e+03
1.0e+04
1.5e+04
2.0e+04
2.5e+04
3.0e+04
3.5e+04
4.0e+04
EstimatedGFLOPS
CompletedTasks
Time HH:MM
Tasks
Aggregated Power (GFLOPS)
New Web Technologies make JavaScript engines
more powerful:
• Web workers
• web socket
• WebGL
• WebCL
35. • Clck
to
edit
Master
0tle
style
Conclusions
• WS-‐VLAM
has
interes0ng
features
(farming,
automa0c
scaling,
hierarchical
workflow,
provenance,
…)
which
proved
to
be
interes0ng
for
a
number
scien0fic
applica0ons
• WS-‐VLAM
harnesses
various
type
of
compu0ng
resources
(desktops,
Grid,
and
Cloud
resources)
• WS-‐VLAM
has
modular
design
which
make
easy
to
adapt/
extend
36. • Clck
to
edit
Master
0tle
style
http://www.science.n/~gvlam/wsvlam/
http://www.commit-nl.nl/