This talk will examine issues of workflow execution, in particular using the Pegasus Workflow Management System, on distributed resources and how these resources can be provisioned ahead of the workflow execution. Pegasus was designed, implemented and supported to provide abstractions that enable scientists to focus on structuring their computations without worrying about the details of the target cyberinfrastructure. To support these workflow abstractions Pegasus provides automation capabilities that seamlessly map workflows onto target resources, sparing scientists the overhead of managing the data flow, job scheduling, fault recovery and adaptation of their applications. In some cases, it is beneficial to provision the resources ahead of the workflow execution, enabling the re-use of resources across workflow tasks. The talk will examine the benefits of resource provisioning for workflow execution.

1. The Interplay of Workflow Execution
and Resource Provisioning
Rafael Ferreira da Silva
18th SIAM Conference on Parallel Processing for Scientific Computing (SIAM PP)
MS10 Resource Management, Scheduling, Workflows: Critical Middleware for HPC and Clouds
March 7, 2018
Funded by the National Science Foundation under grants #1741040 and #1664162

2. https://pegasus.isi.edu
OUTLINE
Introduction Workflow Systems Workflow Execution
Workflow Optimizations Workload Characterization
Compute Pipelines
Scientific Workflows
Scientific Achievements
Basic Concepts
Execution Challenges
Heterogeneous Environments
Fine-Grained Workflows on
HPC Systems
HPC/HTC
Resource Provisioning
Fault-Tolerance
Task Clustering
Workflow Restructuring
2
Extreme Scale
In Transit Processing
In Situ Processing

3. https://pegasus.isi.edu 3
Compute Pipelines
Building Blocks
Compute Pipelines
Allows scientists to connect different codes together and execute their
analysis
Pipelines can be very simple (independent or parallel) jobs or complex
represented as DAGs
However, it is still up-to user to figure out
Data Management
How do you ship in the small/large amounts data required by your
pipeline and protocols to use?
How best to leverage different infrastructure setups
Grids have no shared filesystem while XSEDE and your local campus
cluster has one!
Debug and Monitor Computations.
Correlate data across lots of log files
Need to know what host a job ran on and how it was invoked
Restructure Workflows for Improved Performance
Short running tasks? Data placement

4. https://pegasus.isi.edu
Why Pegasus
4
> > >Automation
Enables parallel, distributed computations
Automatically executes data transfers
Reproducibility
Reusable, aids reproducibility
Records how data was produced (provenance)
Automate
Recover
Debug
Recover & Debug
Handles failures with to provide reliability
Keeps track of data and files
NSF funded project since 2001, with
close collaboration with HTCondor
team
http://pegasus.isi.edu

5. https://pegasus.isi.edu
Scientific Achievements
5
60,000 compute tasks
Input Data: 5,000 files (10GB total)
Output Data: 60,000 files (60GB total)
executed on LIGO Data Grid,
Open Science Grid and XSEDE

6. https://pegasus.isi.edu 6
Advanced LIGO
PyCBC Workflow
One of the main pipelines to measure the statistical significance of
data needed for discovery
Contains 100’s of thousands of jobs and accesses on order of terabytes
of data
Uses data from multiple detectors
For the detection, the pipeline was executed on Syracuse and Albert
Einstein Institute Hannover
A single run of the binary black hole + binary neutron star search
through the O1 data (about 3 calendar months of data with 50% duty
cycle) requires a workflow with 194,364 jobs
Generating the final O1 results with all the review required for the first
discovery took about 20 million core hours
PyCBC Papers: An improved pipeline to search for gravitational waves from compact binary coalescence. Samantha Usman, Duncan Brown et al.
The PyCBC search for gravitational waves from compact binary coalescence, Samantha Usman et al (https://arxiv.org/abs/1508.02357)
PyCBC Detection GW150914: First results from the search for binary black hole coalescence with Advanced LIGO. B. P. Abbott et al.

7. https://pegasus.isi.edu 7
Southern California Earthquake
Center’s CyberShake
Builders ask seismologists: What will the peak ground
motion be at my new building in the next 50 years?
Seismologists answer this question using Probabilistic
Seismic Hazard Analysis (PSHA)
286 sites, 4 models
each workflow has
420,000 tasks
CPU jobs (Mesh generation, seismogram synthesis):
1,094,000 node-hours
GPU jobs: 439,000 node-hours
AWP-ODC finite-difference code
5 billion points per volume, 23000 timesteps
200 GPUs for 1 hour
Titan:
421,000 CPU node-hours, 110,000 GPU node-hours
Blue Waters:
673,000 CPU node-hours, 329,000 GPU node-hours

8. https://pegasus.isi.edu
Enabled cutting-edge domain science (e.g., drug
delivery) through collaboration with scientists at the DoE
Spallation Neutron Source (SNS) facility
A Pegasus workflow was developed that confirmed that
nanodiamonds can enhance the dynamics of tRNA
It compared SNS neutron scattering data with MD
simulations by calculating the epsilon that best
matches experimental data
Ran on a Cray XE6 at NERSC using 400,000 CPU
hours, and generated 3TB of data
Water is seen as small red and
white molecules on large
nanodiamond spheres. The
colored tRNA can be seen on
the nanodiamond surface.
(Image Credit: Michael
Mattheson, OLCF, ORNL)
An automated analysis workflow for optimization of force-field parameters using
neutron scattering data. V. E. Lynch, J. M. Borreguero, D. Bhowmik, P. Ganesh, B. G.
Sumpter, T. E. Proffen, M. Goswami, Journal of Computational Physics, July 2017.
Impact on DOE Science
8

9. https://pegasus.isi.edu 9
Soybean Workflow
TACC Wrangler as Execution Environment
Flash Based Shared Storage
Switched to glideins (pilot jobs) - Brings in remote compute nodes
and joins them to the HTCondor pool on the submit host
Workflow runs at a finer granularity
Works well on Wrangler due to more cores and memory per node
(48 cores, 128 GB RAM)

10. https://pegasus.isi.edu 10
cleanup job
removes unused data
stage-in job
stage-out job
registration job
transfers the workflow input data
transfers the workflow output data
registers the workflow output data
DAGdirected-acyclic graphs
Portable Description
Users do not worry about
low level execution details
abstract
workflow
executable
workflow
transformation
executables (or programs)
platform independent
logical filename (LFN)
platform independent (abstraction)
Basic Concepts

11. https://pegasus.isi.edu
Overview of Workflow Execution Challenges
11
Workflow
Management
System
Workflow
Design
Workflow
Structure
Sequence
Parallelism
Modeling /
Composition
Language
Scheduling / Resource
Provisioning
Architecture
HPC /
HTC
Optimization
Clustering
/ Cleanup
Pilot
Jobs
Interopera
bility
Task
Estimation
Runtime /
Energy /
Disk
Fault-
Tolerance
Task-level
Anomaly
Detection
Workflow-
level
Exception
Handling
Data
Movement
Data
Locality
Network
Proximity
Reconfigurable
Network
SDN
Execution
Monitoring
Provenance
In situ
NVRAM /
Data
Spaces
In
transit
HDFS /
Burst
Buffers
Deelman, E., et al. The Future of Scientific Workflows, The International Journal of High Performance
Computing Applications, vol. 32 (1), 2017.

12. https://pegasus.isi.edu
Input data site
Data staging site
shared
filesystem
submit host
(e.g., user’s laptop)
Compute site B
Compute site A
Data staging site
Input data site
Output data site
Output data site
object storage
Heterogeneous Environments
12
HPC
Cloud

13. https://pegasus.isi.edu
Workflow Optimization: Task Clustering
13
Task Granularity
Low performance of lightweight (fine-grained) tasks
High queuing times
Communication overheads
Solutions
Group into coarse-grained tasks reduces the cost of data
transfers when grouped tasks share input data, and saves
queuing time
Use pilot jobs to dynamically provision a number of
resources at a time
Chen, W., et al. "Dynamic and fault-tolerant clustering for scientific workflows." IEEE Transactions on
Cloud Computing 4.1 (2016): 49-62.
Chen, W., et al. "Using imbalance metrics to optimize task clustering in scientific workflow executions."
Future Generation Computer Systems 46 (2015): 69-84.
Ferreira da Silva, R., et al. "Controlling fairness and task granularity in distributed, online, non‐clairvoyant
workflow executions." Concurrency and Computation: Practice and Experience 26.14 (2014): 2347-2366.

14. https://pegasus.isi.edu
HPC System
submit host
(e.g., user’s laptop)
workflow wrapped as an MPI job
Allows sub-graphs of a Pegasus workflow to be
submitted as monolithic jobs to remote
resources
Fine-Grained Workflows on HPC systems
14
Dreher, M., et al. "In Situ Workflows at Exascale: System Software to the Rescue." Proceedings of the In
Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization. ACM, 2017.
Ferreira da Silva, R., et al. "A characterization of workflow management systems for extreme-scale
applications." Future Generation Computer Systems 75 (2017): 228-238.

15. https://pegasus.isi.edu
Workflow Optimization: Workflow Restructuring
15
Chen, W., and Deelman, E.. "Integration of workflow partitioning and resource
provisioning." Proceedings of the 2012 12th IEEE/ACM International Symposium on
Cluster, Cloud and Grid Computing (ccgrid 2012). IEEE Computer Society, 2012.
Srinivasan, S., et al. "A cleanup algorithm for implementing storage constraints in
scientific workflow executions." Proceedings of the 9th Workshop on Workflows in
Support of Large-Scale Science. IEEE Press, 2014.
Partition the workflow into subworkflows and send
them for execution to the target system
Storage constraints to limit or minimize the
amount of disk space that workflows use on shared
storage systems (use of cleanup jobs)

16. https://pegasus.isi.edu
Resource Provisioning: Workload Characteristics (HPC)
16
487
Total Number of Users
MIRA (ALCF)
78,782
Total Number of Jobs
5,698
CPU hours (millions)
6,093
Avg. Runtime (seconds)
786,432
Total Number of Cores
49,152
Total Number of Nodes
2014
Physics
73 Users
24,429 Jobs
2,256 CPU hours (millions)
7,147 Avg. Runtime (sec)
Materials Science
77 Users
12,546 Jobs
895 CPU hours (millions)
5,820 Avg. Runtime (sec)
Chemistry
51 Users
10,286 Jobs
810 CPU hours (millions)
6,131 Avg. Runtime (sec)
Schlagkamp, S., et al. ”Consecutive Job Submission Behavior at Mira
Supercomputer." 25th ACM International Symposium on High-
Performance Parallel and Distributed Computing (HPDC), 2016.

17. https://pegasus.isi.edu
Resource Provisioning: Workload Characteristics (HTC)
17
COMPACT MUON SOLENOID (CMS)
1,435,280
Total Number of Jobs
AUGUST 2014 OCTOBER 2014
392
Total Number of Users
75
Execution Sites
15,484
Execution Nodes
792,603
Completed Jobs
385,447
Exit Code (!= 0)
9,444.6
Avg. Runtime (sec)
55.3
Avg. Disk Usage (MB)
408
Total Number of Users
15,034
Execution Nodes
476,391
Exit Code (!= 0)
32.9
Avg. Disk Usage (MB)
1,638,803
Total Number of Jobs
72
Execution Sites
816,678
Completed Jobs
9967.1
Avg. Runtime (sec)
Schlagkamp, S., et al. ” Understanding User Behavior: from
HPC to HTC" Procedia Computer Science (ICCS’16), 2016.

18. https://pegasus.isi.edu
Resource Provisioning: Resource Estimation
18
Jobs’ resource requirements
at Mira
Job submission interarrival
times per day
Job submission
interarrival times per hour
Jobruntime
MemoryUsage
DiskUsage
Ferreira da Silva, Rafael, et al. "Online task resource consumption prediction for scientific workflows."
Parallel Processing Letters 25.03 (2015): 1541003.

19. https://pegasus.isi.edu
Network Provisioning
19
We predicted completion times of current and future workflow
tasks and start times for data staging under different scenarios
to provision appropriate compute & network resources
We developed mechanisms to arbitrate and prioritize data
flows from competing workflows by leveraging an advanced
network provisioning technologies, a virtual Software Defined
Exchange (SDX)
Virtualized
SDX
Site 1
OpenFlow
Controller Site 3Site 2
Data Site
ExoGENI Slice with Virtual SDX + HTCondor Pools + Data Site
Dynamic
Connectivity
HTCondor
Pool
HTCondor
Pool
HTCondor
Pool
10-80 20-70 30-60 40-50 45-45 50-40 60-30 70-20 80-10
Relative data flow priorities (Workflow1-Workflow2)
0
200
400
600
800
1000
1200
1400
1600
1800
Avg.transfertimefor"individual"jobs(secs)
Transfer time vs. Flow priorities (Different bandwidths)
Workflow1 on HTCondor pool 1 (vSDX bandwidth = 250Mb/s)
Workflow2 on HTCondor pool 2 (vSDX bandwidth = 250Mb/s)
Workflow1 on HTCondor pool 1 (vSDX bandwidth = 500Mb/s)
Workflow2 on HTCondor pool 2 (vSDX bandwidth = 500Mb/s)
Effect of relative flow priorities on observed
data transfer times for data-intensive
workflow tasks for different vSDX
provisioned bandwidths
Deelman, E., et al. “PANORAMA: An Approach to Performance Modeling and Diagnosis of Extreme
Scale Workflows”, International Journal of High Performance Computing Applications, vol. 31, 2017.
Mandal, A., et al. “Toward Prioritization of Data Flows for Scientific Workflows Using Virtual Software
Defined Exchanges”, First International Workshop on Workflow Science (WoWS 2017), 2017.

20. https://pegasus.isi.edu
Typical Approaches
Task Retries
Task Resubmission
Task Clustering
Checkpointing
Provenance
…
Statistical and
Machine Learning
Linear Regression
Neural Networks
Classification Algorithms
Tree-based Methods
Support Vector Machines
…
Others
Exception Handling
Game Theory
…
Analytical Solutions
Failure Modeling
Markov Chains
Principal Component Analysis
Histograms
…
20
Some Approaches to Handle Faults

21. https://pegasus.isi.edu
Fault-Tolerance: Anomaly Detection
21
Anomaly Detection System
Developed Auto-Regression (AR) based statistical methods to detect
anomalies for single metrics
AR coefficients and degree of AR calculated from training runs AR model
During application execution, use coefficients to generate predictions; when
error exceeds a threshold, generate triggers
Experimented with anomaly detection by injecting artificial, competing loads Integrated workflow monitoring and anomaly detection system
Mandal, A. et al. “Toward an End-to-end Framework for
Modeling, Monitoring, and Anomaly Detection for
Scientific Workflows”, Workshop on Large-Scale Parallel
Processing (LSPP’16), 2016.

22. https://pegasus.isi.edu
Fault Tolerance: Preventing Faults using PID Controllers
22
P
I
D
ΣΣ Process
OutputSetpoint Input+
-
+
+
+
z
0.0
0.5
1.0
1.5
TimeDead time
Raise time
Percent overshoot
Settling time
Steady-state error
DATAFOOTPRINT
PID controller aims at detecting the possibility of a
fault far enough in advance so that an action can
be performed to prevent it from happening
Ferreira da Silva, R., et al. “Using Simple PID Controllers to Prevent and Mitigate Faults in Scientific
Workflows”, 11th Workflows in Support of Large-Scale Science (WORKS’16), 2016.

23. https://pegasus.isi.edu
In transit processing: Burst Buffers in Workflow Systems
23
Automated BB Reservations
BB reservations operations (either persistent or scratch)
consist in the creation and release, as well as stage in and
stage out operations
Transient reservations: needs to implement stage in/out
operations at the beginning/end of each job execution
Execution Directory
Automated mapping between the workflow execution
directory and the BB reservation
No changes to the application code are necessary,
and the application job directly writes its output to
the BB reservation
Placement of the burst buffer nodes within the Cori system (NERSC)
Ferreira da Silva, R., et al. "On the use of burst buffers for accelerating data-intensive scientific workflows."
Proceedings of the 12th Workshop on Workflows in Support of Large-Scale Science. ACM, 2017.
0
2500
5000
7500
10000
1 4 8 16 32 64 128 256 313
# Nodes
MiB/s
BB no−BB
fx.sgt fy.sgt
1 4 8 16 32 64 128 1 4 8 16 32 64 128
0
500
1000
1500
2000
# Nodes
Time(seconds)
BB No−BB

24. https://pegasus.isi.edu
In situ Processing: WMS collocated with the simulations
24
Integration with In situ Processing Systems
DataSpaces
Distributed in-memory data staging framework
Decaf
Dataflow system for the parallel communication of
coupled tasks
ADIOS
Adaptable IO System
Challenges
Efficient Data Exchanges
Tasks and workflow communication to exchange data
User Code Integration
Expertise required for modifying programs
Task Placement
Management of data locality between tasks
Resilience
Increased probability of failures in large resources
Dynamicity
Workflow graph may need to be restructured at runtime
Portability
Specialized hardware, specific libraries
Dreher, M., et al. "In Situ Workflows at Exascale: System Software to the Rescue." Proceedings of the In
Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization. ACM, 2017.
Ferreira da Silva, R., et al. "A characterization of workflow management systems for extreme-scale
applications." Future Generation Computer Systems 75 (2017): 228-238.

25. The Interplay of Workflow Execution
and Resource Provisioning
Rafael Ferreira da Silva, Ph.D.
Research Assistant Professor, Computer Science Department
Computer Scientist, USC Information Sciences Institute
rafsilva@isi.edu – http://rafaelsilva.com
Thank You
Questions?
Funded by the National
Science Foundation
under grants #1741040
and #1664162
http://pegasus.isi.edu
http://analytics4md.org